Inclusion beyond compliance

Why you shouldn’t rely on automated testing

What does Accessibility Testing Involve?

We’ve written previously about the importance and advantages of having an accessible website. It means anyone, regardless of ability, will be able to understand and operate your digital products.  

This is an excellent and important idea, but it leaves organisations asking, ‘How do we know we meet the requirements?’.

The simple answer is you check your digital product against the accessibility standards laid out in the latest WCAG (the Web Content Accessibility Guidelines) to see how many of its success criteria you meet and implement any changes to areas of your product that aren’t yet compliant. 

If you’re still uncertain exactly what that means and the best method to go about it, keep reading for more in-depth information. 

Who needs to think about accessibility testing?

If you’re a public body in the UK, you probably already know that your digital products are required to meet a certain accessibility standard. The legislation which enforces this was designed to ensure that any user can access essential services. 

However, accessible websites shouldn’t just be a goal for those who have a legal obligation to provide them. At JBi, we strongly believe that everyone should try to make their digital products as accessible as they possibly can. As well as being a moral thing to do, there can be commercial and SEO benefits for organisations with accessible websites. 

So, in short, we believe everyone should consider accessibility testing.

What does accessibility testing involve?

Accessibility testing involves checking each page of your website to see if it meets the success criteria defined in the WCAG guidelines.

The WCAG guidelines are lengthy and can be complicated for anyone unfamiliar with the technical makeup of a digital product. 

Therefore, the simplest way to do this is to have your product tested by a third party who can give you an expert overview of where you are not up to scratch and suggest how to implement fixes. 

There are two main testing methods: automated and manual. Both come with advantages and disadvantages, which are explored and compared in the rest of the article.

Automated Accessibility Testing

As their name suggests, automated tests rely on software to automatically scan pages for issues.

Advantages of Automated Testing

There are plenty of positives to using automated testing, including: 

Price

Pricing of automated testing exists on a scale; there are options to suit a variety of budgets. Depending on which you opt for, you may get more or fewer pages included, a more detailed post-scan report or assistance in rectifying identified issues. 

Moreover, some providers tie in accessibility scans in broader CMS software that might also cover SEO, analytics, content management, and privacy and cookie management. Bundling these together in one tool may be a cost-effective solution. 

Speed

The nature of automated testing means it is faster than manual. They can be performed ad-hoc in a matter of minutes. Ideal if you have a tight deadline to meet. 

Drawbacks of Automated Testing

Balancing out the positives, there are, naturally, drawbacks to automated testing. 

Context

Carried out by software, automated testing does not take the situation into account. This means that it will not spot instances where something technically meets the success criteria of the WCAG guidelines but does not provide a good experience for the user, it may not understand certain types of content or catch every issue. 

For example, automated accessibility tests cannot tell if an image description is accurate or properly judge if the flow of content makes sense for someone using a screen reader. 

Thoroughness

Partly due to the lack of context but also because of limitations in the way they are built, automated checkers also miss accessibility issues. A GDS (Government Digital Service) audit of automated tools found that the best ones only found 30-40% of the known problems. Not enough to be confident of passing an official GDS audit. 

One way to counter this is to use multiple checkers, as one may find issues that another misses. However, this becomes more expensive and time-consuming, eroding some benefits of choosing automated testing.

Manual Accessibility Testing

Manual testing involves having real people review your product.

Advantages of Manual Testing

There are several advantages to employing an expert in accessibility to review your product. 

Context 

The main advantage of manual testing is that it allows for human judgement and contextual understanding, which is absent in automated testing. 

For example, a manual tester might question whether headers, links and other content are organised most logically for a user with cognitive disabilities. An automated scan wouldn’t pick up on this. 

Furthermore, many manual testing companies employ users with disabilities to carry out the testing. This is a great advantage as they may spot things other testers would not notice. 

However, it is worth remembering that not all users with disabilities interact with digital products the same way, so this kind of testing is not a catch-all for every accessibility issue.  

Drawbacks of Manual Testing

Time

Of course, manually reviewing a page takes longer than having the code scanned. And, if you are using an external partner to manually review, you will have to schedule your review in advance. Both of these means manual testing has to be planned and allotted a reasonable time for turnaround. 

Price

Taking up people’s time to do manual tasks is almost always more expensive than automation, and accessibility testing is no exception. However, a reviewer with good experience is almost always worth the price. 

Beyond Compliance: JBi’s Recommendation for Accessibility Testing

Do both. It may seem like a fence-sitting answer, but we truly believe it’s the best way to ensure an accessible website.

The first time you undertake an audit, it can be intimidating. You will likely get 100s of items to action, including things you’d never considered could be an accessibility issue. Some will be quick fixes, and others will take more consideration to resolve. 

But, importantly, every scan from then on will be more painless. Structural issues will be fixed, and increased awareness of how to meet different success criteria should lead to more careful management of new pages, content and features, in turn leading to fewer new issues. 

Once you’ve done the initial hard work, regular automated tests should keep your digital products fine-tuned between manual tests; we recommend one of these a year. 

By combining testing types, you unlock the full range of benefits testing gives and ensure your products meet all legal requirements and have the added benefit of a human sense-check to ensure they do more than tick the boxes and provide genuinely accessible experiences to all. 

Want to know more about accessibility

For more accessibility information you can read our blog posts about why accessibility is so important, how accessibility improves UX for everyone and how to begin making your website accessible. And, you can read more about the accessibility standards and guidelines in our article about WCAG and what it means for you.