National Network of Libraries of Medicine

nnlm.gov

Dragonfly

Winters 2003 -- Volume 34, Number 1

Testing a Web Site for Accessibility: Part II

by Drexel Malone
Student, University of Washington Information School

Part I of this study was presented in a literature review of the topic, in a Dragonfly Newsletter article, Winter 2002, Volume 33, Number 1. Part II presents the usability study of participants with visual impairment using the National Network of Libraries of Medicine, Pacific Northwest Region (NNLM/PNR) Web Site. ¹ The parent organization, the National Library of Medicine, coordinates and funds by contract the National Network of Libraries of Medicine through its network of health science libraries.² Their mission is to "ensure that quality health information is accessible to health care providers and consumers" throughout the Northwest region.

Introduction

The NN/LM PNR Web team of three medical librarians and one technology coordinator realized the need for Web site navigation improvements to the organization's Web page by the fall of 2000. People were having difficulty finding things from the Web site. The Web team decided the front page should become more user-friendly.

Pilot tests of new front page designs were conducted with six librarian participants in the summer of 2001. One front page design was chosen and implemented in December 2001. The NN/LM PNR Web site, as a federally funded program, must be accessible for users with disabilities.³ The team decided to do usability testing specifically for users with vision disabilities. Usability testing was designed and conducted by a graduate student from the University of Washington's Information School in June, 2002.

Purpose of the study

This study proposed to determine if blind people could navigate the NNLM/PNR Web site:

Would they understand the wording of the information on the front page?
Would they identify links listed on the front page?
Could they complete tasks that required navigating the front page?
Could they complete tasks that required navigating further through the subsequent pages?
Would the screen reader interpret the front page to allow completion of tasks?
Would the screen reader interpret the headings, text, links, search box, and drop-down and drop-down list boxes, known as combo boxes, used in the organization of the pages?

Methodology

Participants
The study had six participants. They were chosen for their characteristics of having vision impairment (they were completely blind), having computer skills, and having experience with the World Wide Web. All participants worked with computers in some capacity in their jobs at support organizations and institutions for the blind in Seattle, Washington. Four participants taught computer classes to blind people. Ages ranged from 24 to 52. Participants included five men and one woman.

Assistive Technology
Participants all used much the same hardware and software. They used personal computers and monitors at their workstations. In addition, they used Jaws screen reader, version 3.5 or 4.0 to access Web pages. The Jaws application worked in conjunction with a speech synthesizer to create a vocalization of the screen text. All participants used a refreshable Braille display "keyboard" of 80 characters (some were smaller with fewer characters) at their work stations. One participant did not use the Braille display during testing. The Web could be searched without this last piece of equipment; it was most frequently used to check spelling of outgoing emails and to augment the information presented by the speech synthesizer.

Procedure
The researcher studied participants in their own surroundings instead of having the users come to a centrally located laboratory. This technique allowed the participants to use their own computer and adaptive technology equipment in their own work station. This ethnographic technique allows the researcher to note behavioral influences and motivational factors but still allows objectivity to report data free of bias. ⁴ Usability testing is a combination of quantitative and qualitative techniques. The researcher is interested in the responses and reactions of the participants as well as collecting objective data. ⁵

The researcher recorded the progress of the participant as the tasks were completed through filming, documentation forms, and participant comments. Using the same person to conduct all test sessions provided consistency. ⁶ Data was collected by data forms as well as field notes. The post-test questionnaire provided documentation of reactions after the session was completed (Appendix 1).

Participants were told they would be asked to complete tasks on the NN/LM PNR Web site, and that the researcher would be recording their progress. They were asked to "think out loud," giving the observer clues to their decision-making process. Participants were assured the session was testing the Web site and not the person. They were also told the camera would be filming the monitor screen for the most part, and not the participant.

Tasks
Five tasks were compiled to represent both difficult and simple jobs. Simple, or superficial, tasks could be completed from the front page, that is, the information or link would be found on the front page. Examples of simple tasks were: finding the links to "Dragonfly Newsletter" and "PubMed" (Appendix 2). More difficult, or deeper, tasks would lead the participant further into the Web site. Examples of deep tasks were: finding links to "About Us," Tribal Connections," and a "health sciences library in Spokane." Participants tended to complete the superficial tasks quickly and had varying degrees of difficulty with the deeper tasks.

Results

Review of the time for each task revealed that where tasks took longer than four minutes there was some difficulty completing the task. Participants were not required to finish within a certain timeframe. This allowed the researcher to track their progress and follow their strategy. The shortest duration for a task was one minute or less; the longest duration for a task was 15 minutes (Appendix 3).

Of the five tasks, the two superficial tasks were completed in the shortest period, with the least difficulty. All participants had similar experience with these tasks, completing them within four minutes (Appendix 4).

Participants had varying degrees of ease or difficulty with the deeper tasks. There was no one consistent pattern and no one clear method that worked better than another. Nor did one feature consistently defeat participants. The approach taken by each participant in fulfilling tasks varied from task to task and from person to person. The deeper tasks were completed in 5 to 15 minutes.

Specific difficulties included finding no direct links to the Tribal Connections project on the Pacific Northwest Region site and no direct link to "staff." One participant noticed that, while there is a direct link from the National Network Libraries of Medicine Web page, there is no direct link from the Regional Network Libraries of Medicine Web page, from where the link originates. The newsletter was listed in the link as "Dragonfly" not "Dragonfly Newsletter," allowing a screen-reader user to miss the newsletter if he were employing the Links List feature (The Links List feature reads only the one word listed as the title of the link, or "Dragonfly" and not "Dragonfly Newsletter.") One last issue arose when participants "searched the site." The search page displays, first, a MEDLINEplus search box, before the "search the whole site" box. One participant found this confusing, since the link he had chosen had said he would arrive at the 'search the site' location, not a MEDLINEplus search location. Others were not bothered by the MEDLINEplus interruption.

Post-test Questionnaire
The post-test results showed all participants were positive in their overall analysis of the Web site. Participants described the site as user-friendly, accessible, easy to use, and well laid out. They further felt the site was Jaws-friendly, with few graphics and frames. The amount of information available on the site was impressive. Several participants emphasized the main advantage of the site was that there were several ways to reach information or to complete a task.

Difficulties, in general, were unusual terms that were puzzling or pronounced incorrectly by Jaws. LOCATORPLUS, a journal-finding service on the site, was pronounced "lacaturploos." MEDLINEplus, was pronounced "medlinploos," and UW HSL (University of Washington Health Sciences Library) was pronounced "Uh husel" by the screen reader.

Recommendations

Results were surprising in their inconsistency. No tasks on the NN/LM PNR Web site consistently defeated participants while two tasks were consistently completed quickly. One consistency in the study was that the tasks that took longer were all "deep" tasks, requiring moving off of the front page, although not all participants took a long time with deep tasks. The other consistency was that superficial tasks were completed quickly by all participants.

Jacob Nielson, a guru of Web page design, suggests that a small number of subjects in usability tests will clearly indicate flaws in Web page design. ⁷ But this study with blind participants did not corroborate that finding. Difficulties with deeper tasks were not consistently experienced. Perhaps testing more people would eventually provide clear results.

Results can be interpreted to mean that the NN/LM PNR Web site is quite successful for a majority of the participants with vision disabilities as they completed this series of tasks. This study was a valuable tool in determining if a medical information Web site is accessible to people with disabilities. Despite finding few majority results, the study emphasizes the variety of ways blind people approach Web site searching. It also indicates the participants subjectively considered the NN/LM PNR Web site a usable site.

Reference List

1. National Networks of Libraries of Medicine Pacific Northwest Region. NN/LM PNR.[Web document]. Seattle, WA: The University of Washington Health Sciences Library, 1994. [rev. 23 Aug 2002; cited 5 Jan 2003]. <http://www.nnlm.gov/pnr/>.back to article

2. National Library of Medicine. [Web document]. Bethesda, MD: National Institutes of Health, Department of Health & Human Services, 1996. [rev. 27 Jan 2003; cited 5 Jan 2003]. <http://www.nlm.nih.gov/siteindex.html>. back to article

3. Section 508, The road to accessibility [Internet]. Washington D.C. Center for IT Accommodation (CITA); c2000 [cited 2001 Nov 17]. <http://www.section508.gov/>. back to article

4. Chatman EA. The information world of retired women. Westport, CT: Greenwood Press, 1992. back to article

5. Dumas JS, Redish JC. A practical guide to usability testing. Revised ed. Portland, OR: Intellect, 1999.back to article

6. Rubin J. Handbook of usability testing: How to plan, design, and conduct effective tests. New York, NY: John Wiley & Sons, Inc., 1994. back to article

7. Useit.com: Jakob Nielsen's Website [Web document]. Fremont (CA): Nielsen Norman Group [cited 2001 Oct 17]. <http://www.useit.com/>. back to article

Dragonfly, Winter 2003- Volume 34 Number 1
(posted on PNRNews on February 20, 2003)

This publication is funded in whole with Federal funds from the National Library of Medicine, National Institutes of Health, under Contract No. N01-LM-1-3516.

NN/LM | UW HSL | NN/LM PNR | Contact us: nnlm@u.washington.edu | Revised: February 21, 2003

National Network of Libraries of Medicine

nnlm.gov

Dragonfly

Winters 2003 -- Volume 34, Number 1

Testing a Web Site for Accessibility: Part II

URL: http://nnlm.gov/pnr/news/200301/usability2.html