Accessibility testing and recruitment pitfalls

It turned out, they didn’t use a screen reader. They used a voice assistant, but they didn’t use any assistive technology when using apps and websites. And it wasn’t the first time this had happened that week, either. In those interviews I learned a few things about the product we were testing, but not about the use of screen readers or other assistive technology. How could this have happened?

Some context

First, let’s take a step back. Why was I testing specifically with users of assistive tech? With the European Accessibility Act (EAA) going into effect, more and more companies want to make sure their products are accessible—and rightfully so. One step toward accessibility is doing a WCAG audit. This evaluates if the product technically complies with the law. But full technical compliance doesn’t necessarily lead to a good experience.

More than technical compliance

This is where accessibility testing comes into play. At Hike One, we like to do usability tests with people who use assistive technology (because of an impairment or otherwise). The benefits of this are:

• Experience: It becomes clear if, besides technically complying with the WCAG, the product is easy and pleasant to use.
• Unexpected issues: People have their own ways of navigating interfaces and might use a combination of assistive technologies. The combination of those characteristics with your product might lead to unexpected (technical) problems which are not uncovered by just an audit.
• Confirmation and prioritisation: Testing with people can help confirm problems found in audits and prioritize them, by seeing the impact of an issue.

How to do accessibility testing

A good accessibility test requires the same elements as a usability test:

• thoroughly prepare and test your setup
• use a screener to find relevant participants
• be transparent about the study goal and use of recordings and personal information
• put participants at ease, don’t ask leading questions, etc.

For accessibility testing specifically, pay additional attention to consent, location, and recruitment.

The test itself

To start with the obvious: if you’re testing a prototype (compared to a live product), check whether the prototyping tool supports the assistive technology. For example, Figma supports using a prototype with a screen reader to some extent—as long as you pay attention to your layer names and the interactions only rely on buttons and links.

Consent

While consent forms can already be overwhelming for people without an impairment, a printed form in a small font can be completely unusable for people with a visual impairment or those who are low-literate. So make sure that the consent form is in a format that is available and accessible to your participants, and perhaps send it ahead by email.

Location

While you may be used to inviting participants to your office, it can be valuable to let the participant choose the location. At home, they can use specific tools (like their braille reader or a desktop PC).  This makes the product accessible for them, and it allows you to see how they use these tools. Additionally, traveling to your office might pose a problem for some people. Coming to them broadens the pool of possible participants.

If you do invite them to a different location, make sure to ask them what they need beforehand. This way you can make sure the space (and the study itself) is accessible to them.

Recruitment

For recruitment, you might be tempted to search for people with a specific impairment (‘blind people’). Instead, you might want to recruit people who use a specific kind of assistive technology (‘people who use a screen reader’). This can be the better option for a few reasons:

• Assistive technologies aren’t directly linked to specific impairments – for example, not all screen reader users are blind – so you might miss out on valuable participants by only selecting people with a certain impairment.
• An impairment could be considered medical information. Storing such information about a person falls under ‘special categories of personal data’ under the GDPR, which requires explicit permission from the participant.

My recruitment pitfall

So what happened with that test with the voice assistant user from the intro? And to all the other surprising interviews or participants I encountered? As we only asked our recruitment partner to filter on screen reader usage, this is what fell through the cracks:

• Several people used voice assistants such as Siri or Alexa, and confused that with VoiceOver or a screen reader.
• One participant actively used a screen reader, but not to operate (navigate and control) apps and websites. The reason they used it was to read specific pieces of text out loud, which were otherwise hard to read. Alternative texts for images didn’t matter to them, nor did the roles and structure of elements on a page. Things that do matter to users who fully rely on a screen reader.
• Another participant did try out a screen reader once, but didn’t find a use for it. Because our screener wasn’t explicit about the extent to which the participant used their screen reader, they were still included in the study.

Luckily, those interviews served as any other usability test would have, so we still learned from them.

Additionally, the broad scope of recruitment (by focusing on the tool instead of the impairment) lead us to learn about different ways of using screen readers. For example, two participants used their screen reader only sparingly, as an addition to their zoom tool. This meant they ran into different problems from people who rely fully on the text-to-speech representation of the product.

How to recruit the right audience

Throughout this process, we learned a lot about recruitment. If you’re looking for explorative insights about assistive technology usage, a broad screening could work wonders. But if you want to focus on testing actual screen reader usage with your product, pay extra attention to your screener—and consider hiring a recruitment partner.

A better screener

Of course, you could ask possible participants if they have a specific impairment (‘blind people’), but you’ll still run into the problems mentioned above. What to do instead:

• Ask open questions about why they use a screen reader, and which screen reader they use. Filter out the people who use it just to read out texts¹, or those who appear to confuse a screen reader with a different tool, for example.
• Alternatively, you could ask how often they used a tool in the last month, or compared to their computer/smartphone use. This allows you to select only the most frequent users (or the opposite, if that better serves your research needs).

Partners

If you normally recruit through your own platforms or with the help of a recruitment agency, you could also look further. There are agencies who focus on ‘inclusive testing’. They’re used to finding people with a specific impairment or users of assistive technology. Interest groups (such as the Oogvereniging for visual impairments in the Netherlands) are also a useful way to get in touch with your target group.

Conclusion

Accessibility testing should be part of any product development process, and it requires some extra attention if you’re not used to doing it. We hope our findings help you set up your own research. Please get in touch if you need help!

1. Note that the reason might also include medical information (once again a special category of personal data)