Accessibility Testing Best Practices in CI/CD

For years, web accessibility testing has been a hard process for the development teams. It caused the CI/CD process to slow drastically ⌚

This is because of the stigma that all accessibility must be done as a manual test and there is no way to automate it. The good news is, this stigma is not true, but within reason. Automated accessibility can’t catch all issues.

In reality, it can catch the lower third of accessibility issues that exist on a given application. However, using automated accessibility testing solutions has many benefits that include education, better code quality, and accessibility aware developers who are conscious of the content they create before it goes out the door.

Types of Accessibility Automated Testing

There are two main types of automation that are most effective for development teams in a CI/CD environment, to maintain speed and agility while creating an accessible final product.

The first is accessibility testing libraries that work with your unit, integration, or end-to-end test cases such as Axe-Core and PA11y. These libraries can integrate with a development team’s specific testing library, and become a test case for each page, or component that is tested.

The second type of automated testing solution that is best for CI/CD teams is accessibility linters. Linters in general are tools that help improve your code as you write it 💹 Over the past few years, linters have begun to include checks for accessibility issues as you code.

With this power developers can see the accessibility mistakes they are potentially making without having to run any test cases!

Strategies in CI/CD Environments

The problem that teams run into (knowing that the above two accessibility automated solutions are best for CI/CD environments) is what to do with them. Let’s cover some best practices for teams using them!

Ensure Multiple Levels of Test

One of the cardinal sins of teams that use accessibility testing libraries in their automated tests is only testing one level of test. They will either choose to do unit testing (component level) or integration testing, but not both.

Let’s say a development team is creating their application in React. They are using a test automation framework to test their components and an accessibility testing tool to create a test case that checks if the component is accessible.

A developer may fix all the automated accessibility issues, however, that component may STILL have issues once it is placed onto a full-page.

That is why we must ensure multiple levels of test, so we can fix the general issues in unit testing and see what the issues may be when the component behaves with other components on a page.

No Linting Issues

Linters are a proven way to improve code quality and enforce coding standards for development teams. Normally teams that adopt accessibility linting check if their development environment holds them as “soft checks”, or as checks that they don’t enforce in the project as a whole.

The reason for this is the stigma that it adds a lot more checks, when in reality it only adds an average of 15-20.

If you choose to use accessibility as part of your overall linting strategy, treat it like you would the rest of your code quality checks. You can enforce ZERO accessibility linting issues and create a gate on commit or push.

Create a Testing Strategy

One of the most critically forgotten things to do before adding automated accessibility linting or libraries is to create an accessibility testing strategy.

Automated accessibility testing can seem simple at a high level, however, it can add more stress to CI/CD teams. Since it adds another item to their day-to-day work, you want to ensure that it is done as smoothly as possible 👌

Creating a testing strategy can help ease the tension a team may have to add more to their plate. Some strategies may include:

• Testing and fixing only certain impact levels first (critical or serious ones), then moving to less severe issues.
• Only linting on certain accessibility rules to start with.
• Only testing a certain number of critical pages to start with.
• Slowly adding a zero policy (no accessibility violations) on certain components or pages over multiple sprints.
• Initially testing one level of the test, then working to get to more levels (unit, integration).

Going all in and flipping the switch to “We are now doing automated accessibility checks” is doomed to fail. That is why you have to create a testing strategy around it, in order to get the CI/CD teams to better adapt and get the most value out of the testing.

Regression Automated Testing Accessibility

The final best practice is one that development teams may not even think about 💭 There are out-of-the-box testing solutions for accessibility that do generic checks of issues. However, development teams can easily write their own accessibility regression tests. Yes, you read that correctly.

Let’s go through an example: if the development team have terms and conditions components that expands in a click, a developer could easily write an accessibility regression test to see if the aria-hidden attribute is toggled from true to false on open and close.

These tests go above and beyond out-of-the-box automated accessibility tests, and they add so much value to the overall testing suite.

Just by having those tests, a development team can ensure that all the content is properly set for a screen reader user to be able to use properly and understand its functionality.

Conclusion

Learning how to make your content accessible doesn’t have to slow down your day-to-day development process. Using automation to check for accessibility issues is a great way to ensure your developers are creating code that is accessible and concise, while also educating them as well.

It may seem like a big endeavor for CI/CD teams initially, however, using the best practices above will help to ensure that it will become another part of the development.

Using automation is a great way to get the developers on board with accessibility, and can pave the way to full accessibility testing practices in the future! 💫