2006 Conference General Sessions

Go to previous article
Go to next article
Return to 2006 Table of Contents


USABILITY AND ACCESSIBILITY IN COMPUTER ADAPTIVE TESTING

Presenter(s)
Mark Harniss
University of Washington Center on Outcomes Research in Rehabilitation (UWCORR)
Box 357920
Seattle WA 98195
Day Phone: 206-685-0289
Fax: 206-543-5771
Email: mharniss@u.washington.edu

Presenter #2
Dagmar Amtmann
University of Washington Center on Outcomes Research in Rehabilitation (UWCORR)
Box 357920
Seattle WA 98195
Day Phone: 206-685-4181
Fax: 206-543-4779
Email: dagmara@u.washington.edu

We report findings from usability and accessibility testing of the Patient Reported Outcomes Measurement Information System, a computer adaptive testing system (CAT).

We report findings from usability and accessibility testing of the Patient Reported Outcomes Measurement Information System, a computer adaptive testing system (CAT).

 

Many individuals with chronic conditions have disabilities that limit their participation in activities of daily living. Users with disabilities face challenges when using computer systems if those systems are not developed to be universally accessible. Computer systems may be inaccessible to people with disabilities if they provide only one way for users to gain access to, or manipulate information, particularly if the ability to use the technology depends primarily on user vision or hearing. For example, people who have visual impairments cannot read instructions presented only in a visual format; people who are deaf cannot understand content that is only presented aurally; people who are color-blind cannot discriminate between color-coded options; people who have limited use of their hands or arms cannot use a mouse. Web-based or other computer interfaces with inconsistent layout, difficult to recognize graphics, and inaccessible language are problematic for all users, but present significant barriers for people with some kinds of learning and other cognitive disabilities.

 

Accessible computer systems must be compatible with the assistive technology used by individuals with disabilities such as screen readers or screen magnification. In addition, accessibility features may be built-in such as the option for auditory feedback from the information system, or a high contrast option in computer operating software. Or the information may be available on demand, like closed captioning or audio description for video. Accessible web-based information systems or interfaces are those that provide a text equivalent (typically a description) for all non-text elements, such as audio, video, graphics, animation, graphical buttons, and image maps. They also include clear and consistent navigation and adhere to web content accessibility guidelines, like those published by the World Wide Web Consortium’s Web Accessibility Initiative (WAI) or Section 508. Websites designed in this way allow those who cannot see the screen to access the information using assistive technology such as screen readers used by some people with vision impairments.

The goal of this project was to test the accessibility and usability of an online system that is being developed to evaluate patient reported outcomes. The system, titled the Patient Reported Outcomes Measurement Information System (PROMIS), is a form of computer adaptive testing (CAT). It will be used to measure common symptoms such as pain and fatigue in individuals with chronic conditions such as multiple sclerosis, arthritis, spinal cord injury, etc. In CAT, sophisticated probability estimates (derived through Item Response Theory) allow the system to target the questions asked of the user. The questions a user receives are based on the answers given to previous questions.

 

A CAT survey can be administered via a stand-alone or networked computer, a mobile device such as a personal digital assistant (PDA) or cell phone, or interactive voice response system (IVR). For this study, we only addressed the accessibility of a system designed to be implemented on the World Wide Web.

In this study, we conducted usability tests with individuals with visual, physical, and intellectual disabilities to learn about user interface designs that would be both accessible and usable by this population. The current CAT system does not provide auditory information, so we did not include users who are deaf or have limited hearing. Subjects were provided with a user interface that has been initially tested by a team of experts in the field of accessibility.

 

Researchers sat with each subject in a private room for these usability tests. Subjects sat in a chair (or their own wheelchair) in front of a computer monitor to complete the CAT testing. The monitor was mounted on an ergonomic, adaptable desk to accommodate wheelchairs and other physical needs. If subject’s typically used assistive technology (screen readers, alternative input devices, etc.) it was provided.

Subjects were asked to complete four procedures. First, subjects were given written instructions for accessing the CAT online and were requested to use a PIN number to log on to the system. Second, subjects were asked to complete a section of the CAT that requests demographic information. Third, subjects were asked to complete a series of 30 sample multiple choice questions selected from a larger set of questions on the topics of social health. The questions were selected to cover the range of types and formats of questions possible. Finally, subjects were asked to finalize and submit the results of their testing.

 

Since one of the ultimate goals of the overall PROMIS research project is to develop a CAT easily usable by all individuals, even those with low reading skills, we used the Wide Range Achievement Test (WRAT) to provide an assessment of reading level. The WRAT3 is validated for children and adults above the age of 5 years and was administered after completion of the usability tests.

 

During these usability tests, subjects were videotaped. Subjects were encouraged to “think-aloud” about what they were doing and why. In particular, they were encouraged to note interface issues that confused them or were difficult to complete. Researchers also took field notes during the session. After the session, researchers engaged in a “cognitive interviewing” process in which they asked the subjects about specific choices they made

 

In this presentation, we briefly provide an overview of the project and the importance of computer adaptive testing as it relates to patient reported health outcomes. We note the guidance that is available to CAT developers and users regarding accessibility. We then report the results of these usability studies, highlight the accessibility and usability problems identified, and discuss solutions.


Go to previous article
Go to next article
Return to 2006 Table of Contents


Reprinted with author(s) permission. Author(s) retain copyright