Abstract
 

The author presents the methods and results of a year-long evaluation study conducted solely for the purpose of determining disability accessibility barriers and potential solutions for those barriers found in four Web-based learning environments. The methods outlined include computer-based analysis tools and computer-facilitated focus groups and focussed individual interviews. Additionally, the paper includes URLs for several Web resources on accessibility, including a site created by the author based on the results of the evaluation study.

Introduction
 

The recent push toward Web-based distance education has brought with it a promise of ãanywhere and anytimeä and often also including a companion promise of being ãfor anyone.ä However, the fast-paced coding environment of the Web has the potential to exclude the population that may best benefit from Web-based distance education, students with disabilities, thus tarnishing the ãfor anyoneä promise of these distance education technologies and sparking legal debates centered around the Americans with Disabilities Actâs ãReasonable Accommodationsä  for classroom materials (Hinn, 1999). With approximately one in five people in the United States alone having a legally defined disability (Waters, 1997), the opportunity for educational outreach is great.

The idea about creating accessible Web environments for people with disabilities is certainly not a new one. An article that appeared in an issue of The Chronicle of Higher Education in May of 1994 tells the story of a then new graphics-based Web browser called Mosaic that helped cut the blind off from the Internet (Wilson, 1994). Before the advent of Mosaic, the first graphical-mode Web browser, the Internet was accessed by a text-mode (command-line) interface without graphics. The barrier faced by blind users was the extreme difficulty that assistive devices, such as screen readers that read aloud on-screen information or translate the information into Braille, have with being able to interpret graphics-based information such as buttons or pictures ö a problem that still largely exists today.

However, the inclusion of disability accessibility issues in evaluations of Web-based instructional environments has been extremely uncommon despite the urgings of evaluation theorists such as Ernest House to include the viewpoints of the disadvantaged in evaluations (House, 1993, p. 122). With Houseâs concept of social justice in evaluation in mind, a year-long evaluation (February 1997-February 1998) of the accessibility of a variety of web-based instructional environments created at the University of Illinois at Urbana-Champaign was conducted by the author (Hinn, 1997; Hinn, 1998a; Hinn, 1998b; Hinn, 1998c).

Evaluation Questions
 

The primary evaluation questions used to frame the evaluation were:
 
• Are there any features of the specific Web-based courseware package (learning   environment) that are difficult to access by persons with disabilities?
• What are the ways in which accessibility might be improved for the Web-based courseware package (learning environment)?
• Are there any standard HTML features of many Web pages in general which are difficult to access by persons with disabilities?
• What tools are available for checking accessibility for future revisions of the Web-based courseware package (learning environment)?

Participants
 

Eleven university students with disabilities participated in a series of computer-assisted focus groups and individual interviewed conducted over the course of the year. The disabilities of the student participants included Attention Deficit Disorder with a Learning Disability (1), Traumatic Brain Injury (1), Severe Visual Impairments (2), Blindness (1), Cerebral Palsy (1), Muscular Dystrophy (3), Quadriplegia with limited arm mobility (1), and Quadriplegia with no arm mobility (1). During the computer-assisted focus groups and individual interviews, the student participants used the assistive technologies that they would normally use while accessing the World Wide Web along with the Web browser of their choice. The assistive technologies used by the students included Braille output devices, keyboard versus mouse navigation, screen magnification software, screen manipulation software, and screen readers. Web browsers used included versions of Lynx (text-mode browser), Microsoft Internet Explorer, and Netscape Navigator.

Methods
 

There were two primary evaluation methods used in the evaluation. The first, a computer-based method, allowed the evaluator to rapidly analyze the source code of the Web-based evaluands for common accessibility errors. The second, the use of computer-facilitated focus groups and individual interviews, served as a compliment to the source code analysis with feedback from the students with disabilities who participated in the evaluation. The following subsections describe these methods in more detail.

Computer-Based Analysis Tools
 

An analysis of the source code of the individual Web pages in each of the evaluands was conducted using ãBobby,ä a Web-based analysis tool designed to help determine page features that may be inaccessible for person with disabilities. Additionally, each site was also checked using Lynx, the text mode Web browser that many blind students use.

ãBobbyä (http://www.cast.org/bobby) was developed at the Center for Applied Special Technology (CAST) and is a free, automated Web page analysis tool. There are two versions of Bobby which designers and evaluators of Web-based instructional resources can utilize. The original and perhaps most widely used version is Web-based and allows users to quickly check a single page of a Web site for its accessibility for persons with disabilities. In this Web-based version, a user simply enters the URL of the page into the text-entry box on the Bobby homepage. Bobby then analyzes the HTML source code, returning a list of accessibility errors and suggestions to the user. Additionally, users can also specify a browser version or HTML version that they would also like Bobby to analyze so that it alerts them to code that may be incompatible with that particular browser or HTML version.

The second and newest version of Bobby is a standalone Java application . This is particularly useful for Web site designers as users can download the application and then test entire Web sites on their local machine. In addition to the analysis features of the Web-based version, this version of Bobby provides the user with previews of what the Web pages would look like in Lynx. The ability to view page layout in at least a Lynx simulation helps address one of the limitations of Bobby which is that sometimes it is not clear how pronounced a particular accessibility problem might be. An example of this is the use of tables. Tables cannot be viewed in Lynx and when Lynx encounters a table, it tries to come up with an alternative layout. Sometimes the layout will allow the text in a table to be read in an acceptable format that is easily readable and understood. But many times the layout is easily readable, particularly in data tables such as ones similar to a spreadsheet worksheet. Being able to view these pages in Lynx is an advantage for evaluators and designers alike as the Lynx previews give a visual representation for particularly problematic Web sites that they can share with the evaluation clients.

However, if an evaluator does not have access to the entire site or itâs not convenient to download the pages onto their local drives, there are a variety of alternative ways to view a site in Lynx. There are several Web-based Lynx simulations that allow a user to enter a URL in a text-entry box and the page script will return a mock-up of how the page would be viewed to a Lynx user. Two of these simulations include ãLynx-Itä available from Salt Lake Community College (http://www.slcc.edu/webguide/lynxit.html) and ãLynx-Meä available from the University of Alberta (http://ugweb.cs.ualberta.ca/~gerald/lynx-me.cgi). The most ideal way of viewing a site in Lynx is to actually go through the Web site being evaluated using the actual Lynx Web browser, as was done in this evaluation study. The advantage to this is that by working through to site, an evaluator is alerted to features of the site that may not work correctly using Lynx. In this particular evaluation, occasionally a password protection dialog box or a Web-based form used in an online quiz did not work correctly when used with Lynx. If these features had been viewed using a Lynx simulation, these difficulties would not have been evident. Lynx is available from several public telnet sites including one at the University of North Carolina (telnet://public.sunsite.unc.edu ö use ãlynxä as the login) and one in Maryland (telnet://sailor.lib.md.us/ ö use ãguestä as the login).

Computer-Facilitated Focus Groups and Individual Focussed Interviews
 

Several computer-facilitated focus groups and individual focussed interviews with the student participants with disabilities were conducted during the evaluation. There were several purposes for conducting these interviews. The first was to complement the computer-based analysis tools as a way to provide richer, more personalized examples of accessibility issues. A second purpose of the interviews was to help address the limitation of the ãBobbyä tool in particular. Since ãBobbyä is a source code analysis tool, there are many aspects of accessibility that cannot be automatically checked such as whether or not a background image and text color choices that are used for a Web page provide a sufficient contrast for test readability. Finally, the interviews included student participants with a wide range of disabilities that helped to offset the primary emphasis of the computer-based analysis tools on persons with visual disabilities.

The computer-facilitated focus group is similar to the traditional focus groups in that they involve face-to-face interaction that builds on group discussion (Worthen, Sanders, & Fitzgerald, 1998, p. 382).  However, the computer-facilitated focus group differs in that each focus group member uses a computer throughout the focus group interview. The advantage to conducting a focus group in this manner is that participants can directly demonstrate difficulties with a particular Web-based instructional environment and it also serves as a way to help ensure that each participant understands which particular feature another focus group member or the interviewer is discussing. In this evaluation, each participant used a computer that they were comfortable with and that was equipped with any assistive technologies that they needed in order to access Web-based learning environments.

Another difference between traditional and computer-facilitated focus groups is the size of the groups. Richard Krueger suggests in the second edition of his ãFocus Groupsä (1994) text that large focus groups of 10 to 12 participants are no longer seen as necessary, especially with complex topics. He suggests that focus groups with 5 to 7 participants are easier to set up and manage, as well as they allow individuals to have more opportunities to speak (p. ix). However, computer-facilitated focus groups may require groups of no more than 5 participants. As previously mentioned, one of the features of computer-facilitated focus groups is that they allow the participant to directly point out various strengths and limitations of the evaluand to the evaluator, an advantage that often requires a greater amount of attention by the evaluator and fellow focus group members. With participants with disabilities, particularly students who have speech impairments and/or require a greater amount of time to access particular features on a Web page with their assistive technologies, conducting smaller focus groups can definitely be more manageable. There is a tendency for participants to move ahead of others when working with Web-based evaluands, especially when there is a participant that requires more time to navigate through the site than the others, creating a potential for chaos in a larger group. So an advantage of conducting small focus groups is that the evaluator can concentrate more fully on each of the participants, as well as reduce the potential for confusion.

However, a few of the participants, were unable to participate in even in a small (2-3 person) computer-facilitated focus group. This was due to highly individualized assistive technology that these participants required to access to Web pages of the evaluands, technologies that were more specialized than in the assistive computer labs on the University campus. In these cases, individual focussed interviews that followed the same interview guide used in the focus groups were conducted. A clear limitation in conducting individual focussed interviews versus focus groups is the lack of idea sharing amongst participants. But as one student participant in this evaluation pointed out, it that it doesnât make a lot of sense to conduct a computer-facilitated focus group when the participants canât use the computers in the lab. By using computers and assistive technologies that a student is familiar with, the evaluator can be more certain that a difficult to access Web page feature is due to an access issue alone rather than an access issue and the studentâs use of unfamiliar, inappropriate, or no assistive technologies. It should be noted that the cost of constantly updating assistive technologies sometimes prohibit absolute certainty that either the student at their personal workstation or a group of students in an assistive computer lab is using the most ideal assistive technologies to access Web-based learning environments.

There are a number of benefits to conducting participant individual interviews and focus groups facilitated by participant use of the Web-based courseware tool. First, observing the student(s) as they move through each of the Web pages and their corresponding features can reveal strengths and limitation of the environment that may not have been addressed in the original interview guide. These observations can also help shed light on how well certain assistive technologies that the student(s) might be using to interact with the Web-based learning environment, particularly certain components of the environment such as interactive forms or particular graphics. As both participant(s) and interviewer discover certain strengths or limitations of the environment, the focus of the interview questions can delve a bit more deeply into the heart of a specific strength or limitation.

For instance, a student might make a comment that a particular feature, such as the ãsubmitä button for a form on a Web page, does not seem to work. After a closer observation of the participant trying to interact with the feature in question, the evaluator can inquire further about the issue at hand. This can help the evaluator understand whether or not the problem is due to user error of some sort, unclear instruction on the part of the Web-based environmentâs developer, or is perhaps due to a difficulty that a particular assistive technology has with interacting with the feature in question. In the focus group, the evaluator can also take this opportunity to check with other participants as to whether or not the feature in question is difficult for them as well. The group can work together to try to determine whether or not the access issue is a global one (e.g., true for all Web pages that contain the feature) or more local one (e.g., true for the feature as implemented on this page in particular but necessarily true across all Web pages containing the feature).  An example of this from this evaluation study would be a quiz submission button that was unique to one of the Web-based instructional environments that at first appeared to be a more global issue. When examining another Web-based quiz in a different learning environment that did not present a problem for the same focus group participants, it turned out that the problematic submission button was a problem local to that particular learning environment. Additionally, the uniqueness of a particular access issue can also enter the group discussion where group participants can discuss their feelings about whether the feature being discussed is problematic for students with one type of disability in particular or whether it is also problematic or even an advantage for students with other types of disabilities.

As with this evaluation, it is sometimes the case that in the group discussions, another student will have a solution or work-around for the problem that the student who first made note of the problem was not aware of. This assistive interaction amongst the participants is encouraged, as it is the opinion of the author through observation that this kind of helping environment can be quite beneficial to the participants. Similarly, the evaluator is encouraged to share his or her computer expertise or knowledge of potential solutions to access barriers with the participant.

The following sections comprise of the more generalizable results from the study. It should be noted that it is not a complete listing of all accessibility issues. For a more complete listing of accessibility issues, please see the World Wide Web Consortiumâs Web Accessibility Initiative homepage (http://www.w3.org/TR/WD-WAI-PAGEAUTH/).

Limitations
 

There were several limitations that were identified by the student participants as well as the computer-based analysis tools. The most common limitations and suggested solutions are discussed in the following subsections.

Lack of Alternative Text for Images, Imagemap Hotspots, and Applets
 

When alternative text (ALT tag) is not included within image tags, for imagemap hotspots, and/or within Java applet tags, users who use text-mode browsers such as Lynx and/or use screen readers will be unable to know whether the item was important or just decorative. In these cases, students who rely on these technologies, such as students who are blind or who have severe dyslexia, will only get placeholders such as [image] or [link] and no way of knowing what the image might have been or what the link might be prior to following it. In this evaluation, this was the most common accessibility issue across all of the four evaluands. However, adding alternative text is probably the easiest evaluation issue to fix. The code to include alternative text for each item appears below:

• Alternative text for graphics: <img src="image_name.jpeg" alt="description of image">
• Alternative text for imagemap hotspots: <area shape="rect" coords="50,50,300,30" href="file_name.html" alt="description of link">
• Alternative text for Java applets: <applet code="applet_name.class" alt="description of what the applet is">

However, it should be noted images and applets may need more explanation than just a title and brief description placed in the ALT tag. Therefore it was recommended that, in addition to providing short alternative text descriptions, descriptions, in as much detail as necessary, of the function and interpretation of any complex graphics (such as charts or graphs) and applets should accompany the image or applet in either nearby text or in an accessible alternative on another Web page.

Forms Usage
 

In each of the Web-based environment examined in the evaluation, each used Web forms for online quizzes as well as chat boards. However, several of the students in the study needed to use an extraordinary amount of physical effort in order to use these features. This was particularly the case for the participant who was blind who could not access a quiz using Lynx and was able to use their screen reader with a graphical-mode browser. When trying to answer quiz questions, being able to read each question and then enter the answer proved to be a difficult proposition. When this student used the tab key to move from question to question, the screen would often scroll past the question. When the student tried to get the screen to scroll back in order to hear the question, the placement of the cursor within the answer entry box was then lost. The student then was required to guess where the entry box was on the screen in order to submit their answer.

Web forms, such as those that might be used for an online quiz or message board, should be designed to facilitate independent keyboard navigation as much as possible. Students who have disabilities that limit their arm mobility, such as those with cerebral palsy and muscular dystrophy, often also rely on keyboard navigation. These students move the cursor using the keyboard accessibility features in their operating system. Despite the usefulness of the keyboard accessibility features, they still require quite a bit of manual dexterity to use for navigating forms, especially when the layout of a form does not facilitate moving from answer entry field to answer entry field. It was recommended to the developers of each site in the study that they test their quizzes using only the keyboard to determine the ease of navigation in this manner. However, because of the difficulties that the participant who was blind had, it was also noted that alternative means for quiz submission should also be supported when needed and course instructors should be made aware of possible accessibility issues with Web-based forms for course planning purposes.

Frames Usage
 

The use of frames to organize information in Web sites has been a problem for people using browsers that do not support frames. For a long time, people who used the text-mode browser Lynx could not access sites with frames at all. More recent versions of Lynx now support the use of frames by providing a listing of the pages that comprise the framed site for the user to choose from. This has resulted in the unfortunate tendency, as was evidenced in a few of the sites examined in this evaluation, for Web designers to assume that an alternative non-framed site no longer is needed and omitting the <NO FRAMES> instructions altogether or simply including remarks instructing the user to upgrade to a newer version of Internet Explorer or Netscape Navigator within the <NO FRAMES> tag. In one case, the designers of one of the sites used in the evaluation had created a "help" section for their instructional environments, one that contained eight separate frames without a <NO FRAMES> alternative. When the blind participant encountered the help pages, they were presented with the listing of all eight pages, each represented by the frame name assigned in the HTML code as opposed to their specific URL. Unfortunately, the designers had also not taken care to assign a frame name that might be intuitive to the user navigating with a text-mode browser and was presented with eight pages entitled "top," "middle," "right," "logo," et cetera. To make matters worse, several of the pages were further broken down and by the time the student found what they were looking for, they no longer had any idea where they were due to the recursive design of the frames. Had the designer included a <NO FRAMES> page that might have led the user to an annotated set of alternatives links or even used frame names which more intuitive names such as "navigation bar" and "main frame window," much of this students frustration would have been greatly lessened.

Additionally, the Web-based instructional environments that used frames in this study presented unexpected problems for the one participant who was Quadriplegic without any arm mobility. This student, as do many other students with disabilities that affect their arm mobility, used a speech-input device to navigate through each of the Web-based learning environments. However, when this student encountered areas that utilized frames, they needed to try to force a particular frame into focus (e.g. activate one frame over another) so that they follow any of the links using their speech-input device. So while it was possible for this student to access and use the framed site, it provided the student with a few slow and tedious extra steps. By providing users no only with a <NO FRAMES> alternative within the HTML code but also by providing them with a link to this no frames alternative, either from the top of one of the frames or from a lead page that gives the users a choice in which version they wish to browse, this problem also could have been easily avoided.

Graphical Icons
 

Students with impaired vision without total blindness will often use a graphical-mode browser such as Netscape and Internet Explorer but will raise the font size to an extremely large size, as was the case with several of the participants in this study. However, there were many times where a graphic icon with text labels as a part of the icon were used as a navigational device in a particular learning environment. Since graphic icons cannot be enlarged in the way that text can in a Web browser, these students were either unable to differentiate one icon from another at all or were forced to sit at an extremely close proximity to the screen in an attempt to try to navigate through the site. On the other hand, these navigational devices were noted as an advantage by the student with a learning disability, noting that the graphical cue helped them more than a simple text listing of the pages contained in the site. With this in mind, it was recommended to the developers of the sites that they include text alternatives for each graphical navigation device (such as a cluster of navigational icons or an image map) in addition to the graphical navigation device.

Tables Usage
 

The use of tables on Web pages is another accessibility issue that holds an advantage for some users. There are times when the ideal method for visually organizing information is through the use of a table. In fact, the use of tables is sometimes the best organizational layout for students with cognitive disabilities. However, for students who require the use of screen readers, the text will become jumbled, as screen readers will typically read across columns versus one cell at a time. Another issue is for students with impaired vision without total blindness. When these students use graphical-mode browser by raising the font size in their browser, the text in one table cell will often end up overlapping the text in another table cell, returning a confusing layout as was noted by several of the study's participants. It was recommended the site developers that they link to a non-tables version of the information found in each table.

Browser-Specific Code
 

From time to time, code was included in one of the evaluands that did not work with text-mode browsers. One example of this was where a graphical icon was used for a form submission button versus the traditional gray "submit" button. The code underlying this feature did not work at all for the blind participant who was using Lynx and the code only worked with the latest version, at the time, of Netscape Navigator and Microsoft Internet Explorer. Unfortunately, this was not evident until after the student had completed the entire mock quiz used for the purpose of the student and the student was then unable to submit the quiz.

Another example of browser-specific code usage was when the same student tried to access one another one of the Web-based instructional environments. As with the other environments, this site had a password entry dialog box. However, the coding behind this particular password dialog box was browser-specific and the student was unable to access the site at all ö a pretty severe access barrier. It was recommended that the developers try to avoid situations such as these in an actual course situation by testing each essential feature that uses browser-specific code with a variety of browsers and prepare a non-browser specific alternative. Additionally, it was also recommended to the developers that they each prepare a FAQ page that outlines any known accessibility issues related to browser-specific code as well as presents accessible alternatives as well as a person to contact for solutions to any problems that have not yet been noticed and/or addressed.

Strengths
 

Perhaps the greatest strength found in this evaluation was the simple fact that the environments allowed for the course materials being online. Having course resources available in electronic format can serve as an advantage for many students with mobility and visual disabilities. As one focus group member with Quadriplegia stated: ãFor me·having things around on the Net is a lot easier to read materials because I donât have any hand movement. So in other words, using a book and so forth to look things up·itâs a lot harder. Whereas having it on the computer, I can just sit there and go ahead and work through it and get everything from the page. But I wish that I had a lot more classes that had [materials] on the Netä (Hinn, 1997). For some students, Web-based instruction may be the only way that some students can independently access courses and course-materials ö something that is a powerful reminder of the need for accessible online distance education.

Beyond the Evaluation Study
 

In addition to addressing the needs of students with disabilities in this evaluation in order to increase the utility of the evaluands, the evaluation also was concerned with maximizing the dissemination of the evaluation results to include the greater Web design community. With this in mind, a Web site was created by the evaluator called Access.Edu (http://lrs.ed.uiuc.edu/access) as a way to try to reach out to Web designers who may be looking for more information about the creation of Web sites that are accessible to persons with disabilities. In addition to information about some of the more common access issues from this evaluation, the site contains links and information about companies, organizations, and Web accessibility tools. The site also contains a Web site design tutorial called ãConsidering User Differencesä that was designed to help novice Web page designers understand the accessibility issues surrounding a designerâs choices of color and backgrounds in Web pages through visual and interactive examples. This tutorial also created by the evaluator, focuses on users with reading disabilities such as dyslexia, color-blindness, and visual impairments.

Due to the academic setting in which this evaluation study was conducted, a luxury existed where the evaluator was free to look solely at disability accessibility issues in the four Web-based learning environments that were the focus of the evaluation. Certainly most evaluators of Web-based instruction will not find themselves in quite the same circumstances. However, it is hoped that the methods used and results of this evaluation will provide evaluators with a place to begin thinking about the inclusion of disability access issues, as well as access issues in a more general sense, in their own evaluations of Web-based instruction. It is also hoped that the information in this paper will provide designers of Web-based instructional environments with a few ideas for creating more accessible Web-based learning environments in the future.

Center for Applied Special Technology (1998, August 26). Welcome to Bobby 3.0. [Online]. Available: http://www.cast.org/bobby

Cunningham, C., & Coombs, N. (1997). Information access and adaptive technology. Phoenix, AZ: Oryx Press.

Hinn, D. M. (1997). Evaluation of the Educational Psychology 390 Web site with regards to accessibility for persons with disabilities. Unpublished final technical report, University of Illinois at Urbana-Champaign.

Hinn, D. M. (1998a). Evaluation of CyberProf with regards to accessibility for persons with disabilities. Unpublished final technical report, University of Illinois at Urbana-Champaign.

Hinn, D. M. (1998b). Evaluation of Mallard with regards to accessibility for persons with disabilities. Unpublished final technical report, University of Illinois at Urbana-Champaign.

Hinn, D. M. (1998c). Evaluation of the Virtual Classroom Interface with regards to accessibility for persons with disabilities. Unpublished final technical report, University of Illinois at Urbana-Champaign.

Hinn, D. M. (1999). The impact of visual information in Web-based instruction on students with disabilities. In R. E. Griffen (Ed.), Selected Readings of the International Visual Literacy Association. State College, PA: International Visual Literacy Association.

House, E. R. (1993). Professional evaluation: Social impact and political consequences. Newbury Park, CA: Sage.

Krueger, R. A. (1994). Focus groups: A practical guide for applied research. Thousand Oaks, CA: Sage.

Waters, C. (1997). Universal Web design: A comprehensive guide to creating accessible Web sites. Indianapolis: New Riders.

Web Accessibility Initiative (1998, September 8). WAI accessibility guidelines: Page authoring. [Online]. Available: http://www.w3.org/TR/WD-WAI-PAGEAUTH/

Wilson, D. L. (1994, May 4). Computer access for the disabled: Growing use of graphical devices in computing is cutting some people off. The Chronicle of Higher Education, A25.

Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1998). Program evaluation: Alternative approaches and practical guidelines. White Plains, NY: Longman