Combination of automatic and manual testing for web accessibility

Abstract

Web accessibility is an indispensable medium for online communication and digital inclusion nowadays. With the recent adoption of the Web Accessibility Directive making the Internet resources accessible has become a legal obligation and strikes a need for more detailed and reliable ways of web accessibility evaluation of the websites. Throughout the years, many tools have been developed for testing web accessibility as well as a plethora of metrics that are expected to convey the results. Unfortunately, in most cases the findings appear to be incomplete since the studies rely only on one testing method, i.e., automatic or manual. The study has set itself a goal to contribute with knowledge to solving three research questions. First, how to combine results from automated and manual evaluation of web accessibility? Second, how to express the integration results in a quantitative manner? Finally, what is the impact of the dynamic content on the integration results when the content of the website is frequently updated and personalized? This thesis proposes a novel approach to integration of manual and automated accessibility testing, where the results of the evaluations are combined on the basis of accessibility guidelines. Additionally, a quantitative metric – Union Score, together with a graphical visualization called Accessibility Pie Chart, are propounded, as the means for expressing the outcomes of the accessibility evaluation with use of the combined approach. The research has been grounded on the mixed-method approach and embedded the findings of the conducted interviews into a quantitative study. In order to find emphirically the most suitable method for combining manual and automated testing, fifteen web pages selected from two websites were chosen for evaluation with two testing tools: WTKollen Checker and WTKollen User-Testing Tool. The findings of the analysis show that WCAG 2.0 may serve as a bridge between manual and automated evaluation outcomes and result in an increased coverage of the Success Criteria. The proposed metric has been preliminarily validated with regard to its application for benchmarking purposes and supplemented with a graphical way of presenting accessibility testing results. Furthermore, it is concluded that the suggested integration approach can be deployed. Yet, the challenge of dynamic content evaluation requires more research attention. The study has contributed to the current state of knowledge about web accessibility evaluation and the results are expected to be used for implementation of the novel approach. For the future paths, a more extended study on the proposed metric’s properties is advised. Also, the importance of further research in the area of dynamic content evaluation is highlighted.