Review: Guidelines are only half of the story: accessibility problems encountered by blind users on the web

In sum the usability study, Guidelines are only half of the story: accessibility problems encountered by blind users on the web (henceforth referred to as "the study"), presents several usability issues as accessibility issues. In the short term one issue may warrant WCAG WG action. The study expands the concept of accessibility to include usability. Expanding accessibility to include usability has previously occurred and is currently occurring in other venues including the United States Department of Education, Office for Civil Rights (OCR) resolution agreements. This raises broader questions such as is a tighter integration of usability and accessibility needed? Should the World Wide Web Consortium (W3C) Web Accessibility Initiative (WAI) consider furthering efforts in the area of "Usable Accessibility" and User Centered Design (UCD)? Are broader design principles in order? These longer term questions may deserve contemplation if the Working Group has not already considered and come to consensus on them.

Literature Review

The study was discussed in an article by Detlev Fischer, on the WebAIM mailing discussion list, and on a SitePoint Form.

Prior Discussion
Source Synopsis
Fischer, Detlev. Methodological flaws put question mark on study of the impact of WCAG on user problems, 2012. Discounts most of the study's data, metrics, analysis and conclusions, except the issue of clearly identifying links.
Gunnarsson, Birkir R. et. al. Thread: Guidelines are only half of the story: accessibility problems encountered by blind users on the web, WebAIM mailing list discussion, 2012. General agreement that the 6 issues, which the study called out as not covered in WCAG 2.0 were not something that accessibility guidelines could or should specifically address.
Mallory van Achterberg. Research shows adhering to WCAG doesn't solve blind users' problem, SitePoint forum, 2012. Found that issues presented in the study were not specifically accessibility related but rather usability related and outside of WCAG's scope. The author suggests that if usability criterion were to be increased in WCAG, WAI's definition of "accessibility" would require broadening.

Confusing Usability Issues with Accessibility Issues

Accessibility and usability are closely related and often conflated, as they both can improve satisfaction, effectiveness, and efficiency.

In the WebAIM thread Andrew Kirkpatrick illuminated, of the specific issues raised in this particular study, 6 are not covered in WCAG 2.0. In that same WebAIM thread Léonie Watson comments how it is difficult to map these issues to WCAG. It is difficult to map them because they appear to be usability issues.

ISO 9241 explains usability as the "extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use." Specifically this means designing systems or products that have the following characteristics:

Mapping Issues to Usability Characteristics

Reported Issues Mapped to Usability Characteristics
Number Issue Usability Characteristic
1 Content found in pages where not expected by users Efficiency
2 Content not found in pages where expected by users Effectiveness
3 Pages too slow to load Effectiveness
4 No alternative to document format (e.g. PDF) Effectiveness
5 Information architecture too complex (e.g. too many steps to find pages) Efficiency
6 Broken links Error tolerance

One Issue May Warrant Action Short Term

The issue of "Content found in pages where not expected by users" could possibly be addressed by reinstating WCAG 1 Checkpoint 13.1. As Detlev Fischer stated in Methodological flaws put question mark on study of the impact of WCAG on user problems:

The study's conclusion that the sufficient techniques for SC 2.4.4 Link Purpose are in fact not sufficient at least for the group of blind users strikes me as valid. WCAG 1.0 recommended, in Checkpoint 13.1: "Clearly identify the target of each link. Link text should be meaningful enough to make sense when read out of context." WCAG 2.0 has relaxed this requirement by allowing authors to provide the purpose in the link's context. For blind users who rely on generated indices of headings, WCAG 2.0 is clearly a step backwards. As the lack of descriptive links will also contribute to other frequent problems such as "Content not found in pages where expected by users", it may indeed be worth rethinking SC 2.4.4 in future work on WCAG.

Usability Does Not Equal WCAG Conformance

A Web site can conform when evaluated against WCAG 2.0 yet be difficult to use for both people with disabilities and people without disabilities. Issues called out in the study (Content found in pages where not expected, content not found in pages where expected, pages that load slowly, no alternative to document formats, complex information architecture, and broken links) will likely affect all users not only people with disabilities.

Shawn Henry points out in Accessibility in User-Centered Design: Usability Testing that usability testing,

...doesn't evaluate conformance to accessibility standards.

Others have Expanded Accessibility to Include Usability

The authors of the study are not alone in combining accessibility and usability. Brian Kelly introduced the concept of ‘holistic approaches’ to accessibility in 2004. This approach to accessibility takes into account factors such as usability.

Much more recently the definition of "accessible" has been expanded to include usability in United States Department of Education, Office for Civil Rights (OCR) resolution agreements. The OCR and the U.S. Department of Justice (DOJ) share enforcement responsibility for academic and public accommodations under the Americans with Disabilities Act and its 2008 Amendments and Section 504 of the Rehabilitation Act. These agencies have the authority to conduct a compliance audit or to initiate an investigation in response to a complaint, which can be filed by anyone. These agencies will often seek to enter into a resolution agreement with the subject institution in lieu of conducting an investigation and seeking sanctions or bringing a lawsuit. The OCR has begun to use the following definition:

"Accessible" means a person with a disability is afforded the opportunity to acquire the same information, engage in the same interactions, and enjoy the same services as a person without a disability in an equally effective and equally integrated manner, with substantially equivalent ease of use. The person with a disability must be able to obtain the information as fully, equally and independently as a person without a disability. Although this might not result in identical ease of use compared to that of persons without disabilities, it still must ensure equal opportunity to the educational benefits and opportunities afforded by the technology and equal treatment in the use of such technology.

Sources: South Carolina Technical College System Resolution Agreement (PDF), University of Cincinnati Resolution Agreement (PDF), Youngstown State University Resolution Agreement (PDF)

Usable Accessibility

The study is titled "Guidelines are only half of the story: accessibility problems encountered by blind users on the web." The first half of this title is significant as it expands accessibility to include usability, which the authors conclude is the other half of the story.

It is significant to recognize that usability is an important aspect for people with disabilities. Consciously addressing "Usable Accessibility" may help clarify the difference between what meets WCAG and what is actually usable by people with disabilities. The W3C Web Accessibility Initiative (WAI) explains:

"Usable accessibility combines usability and accessibility to develop positive user experiences for people with disabilities. User-centered design processes (UCD) include both techniques for including users throughout design and evaluation, and using guidelines for design and evaluation. UCD helps make informed decisions about accessible design. Thus UCD is necessary to improve accessibility in websites and web tools...The goal of web accessibility is to make the Web work well for people, specifically people with disabilities. While technical standards are an essential tool for meeting that goal, marking off a checklist is not the end goal. People with disabilities effectively interacting with and contributing to the Web is the end goal. To make the Web work well for people with disabilities, designers and developers need to understand the basics of how people with disabilities use the Web. Following UCD to involve people with disabilities throughout design processes and involve users in web accessibility evaluation helps design solutions that are effective for users and for developers."

The "Guidelines are only half of the story" study concludes by stating:

The results of this study indicate that it is time to move away from the problem-based paradigm for web accessibility, where our primary goal is to eliminate problems encountered by users...Taking a lesson from usability research, web accessibility research must define a much broader set of design principles, based on user data, that focuses on the use of the web by people with disabilities - not just on the problems they encounter. Once those design principles are clearly understood, only then can we look at proposing rules and heuristics that web developers can apply to evaluate their success in creating websites that people with disabilities can use well. This new paradigm will help us to discover the second half of the accessibility story.

Conclusion

Short term the Web Content Accessibility Guidelines Working Group (WCAG WG) may wish to consider the one issue that may have a WCAG implication.

Longer term the WCAG WG may want to contemplate the broader questions such as is a tighter integration of usability and accessibility needed? Are broader design principles in order? And should WAI consider furthering efforts in the area of "Usable Accessibility" and UCD? What is in or out of scope?

Definitions

Effectiveness
The accuracy and completeness with which users achieve specified goals. - [ISO 9241]
Efficiency
The resources expended in relation to the accuracy and completeness with which users achieve goals. - [ISO 9241]
Satisfaction
Freedom from discomfort, and positive attitudes towards the use of the product. - [ISO 9241]

References