logo logo

Integrating Accessibility Tests With TestProject and aXe

main post image

The advantages of using the TestProject OpenSDK for Python have been widely documented since its release, but did you know you can integrate aXe accessibility tests with TestProject and take advantage of the provided Reporting Dashboard?

In this article, we will look at the very simple steps required to report aXe test results in TestProject. But before we get started, let’s look at aXe.

aXe is a web accessibility testing tool that validates web pages against the Web Content Accessibility Guidelines, a list of accessibility standards that web & mobile applications should abide by ensuring accessibility to the many millions who suffer from disabilities. aXe can be used as a browser plugin but there are also many open source packages/libraries which enable testers to incorporate aXe tests into their automation suites alongside browser automation tools such as Selenium. In this demo, we’ll be using aXe with Python.


First, we’ll need to install the following packages

  • TestProject Python SDK (pip install testproject-python-sdk)
  • Pytest (pip install pytest)
  • aXe (pip install axe-selenium-python)

In addition to installing these packages, we also need to add our TestProject Dev Token as an environment variable inside our project. The token can be retrieved from the Integrations tab of the TestProject portal.

Writing the Test

Now that the setup is done, we can jump straight into the script. For the purposes of this tutorial, we’ll be using a very simple script that simply reports any accessibility issues flagged on the registration page of this test todo application.

The first test steps

We can use Selenium to navigate to the page in Chrome and then we can inject aXe to check the accessibility. This will produce a report detailing all recommendations based on the criteria set out in the Web Content Accessibility Guidelines (WCAG).

We will be adding an assertion that will set the test to passed or failed depending on the number of accessibility issues found. If none, the test will pass, however, if one or more issues are identified, the test will be marked as failed.

To start, let’s create a new Python file in our project called test_accessibility.py. Now inside our file, we can add our import statements. We only need 3:

from src.testproject.sdk.drivers import webdriver
from src.testproject.decorator import report_assertion_errors
from axe_selenium_python import Axe

The first imports the TestProject Python SDK, and the second one allows us to record the status of the test based on the result of our assert statement. The last allows us to run our accessibility test using aXe.

Next, we can define our test function and navigate to our application under test. If you know selenium, you’ll be familiar with the following steps:

def test_accessibility():
    driver = webdriver.Chrome()

OK, great, this will get us to our registration page. Now to run aXe. The code we’re using here is taken direct from the usage section of axe-selenium-python on PyPi. It’s just 3 lines of code and its super straight forward.

axe = Axe(driver)
results = axe.run()

In this code we initialize aXe, inject axe-core into the webpage and then run it, storing the output in a variable named results.

If you want to see what the output looks like you can add the following line to your test:

axe.write_results(results, 'results.json')

This will output the accessibility results to a json file inside your project directory. You’ll see that the json file is split into sections. The one we’re interested in is violations. This contains the list of accessibility issues aXe has identified.

Adding the validations

So far, we’ve navigated to our page and run our accessibility test, now we need to add our assertion and report the results in TestProject. In our assertion we’re going to count the number of ‘violations’ reported in the aXe results. This can be done with the following line of code

assert len(results["violations"]) == 0, axe.report(results["violations"])

The first part, before the comma, checks if the number of violations in the results is equal to zero. If not zero then the assertion will fail. The second part will output the violations in the failure report.

Now, if we left our script as it is our test will always show as passed in TestProject, even if there were violations. To make sure assertion failures are correctly reported in TestProject, we need to add the following decorator above our test function, like so:

def test_accessibility():

Cool, almost there 👌 We just need to quit the browser, this is important as it is how TestProject knows to record the test results.


OK – so our full test should look like this:

from src.testproject.sdk.drivers import webdriver
from axe_selenium_python import Axe
from src.testproject.decorator import report_assertion_errors

def test_accessibility():
    # Navigate to Application
    driver = webdriver.Chrome()
    # run axe
    axe = Axe(driver)
    results = axe.run()

    # Output results to file - optional line
    axe.write_results(results, 'results.json')
    # Check the number of violations is zero. If not zero fail assertion
    assert len(results["violations"]) == 0, axe.report(results["violations"])

    # Quit Driver

Running the Test

Now that we have our test, let’s run it using pytest. From inside our project, we can just call pytest via the terminal.

The results in the terminal show that the test has failed and that 4 accessibility violations have been reported:

Run tests in terminal

Test results

Now let’s switch over to the TestProject Platform to see what it looks like there on the reporting dashboards. If we hover over the reports tab we can see that our test has failed:

Accessibility tests report in TestProject

If we drill down into this test, we can see which step has failed:

Accessibility tests results

Clicking on the failed step will also show the violation details. This is not formatted, but you can always go back and consult the results.json file which gets output in the Python project directory.

Accessibility violations



And there you have it: Accessibility test results in TestProject,  in just a few lines of code. Good job! 💪 The TestProject OpenSDK supports the pytest and unittest frameworks in Python, so anything you can do inside these frameworks with Selenium can be reported here. It’s a great way of keeping your results together and utilizing the power of TestProject reporting 📊

Well, that’s a wrap so as always Happy Testing! 😎


About the author

Ryan Howard

Ryan is a Test Engineer and founder of How QA Test consultancy with 10 years experience in the software industry. Ryan loves to talk testing and is the co organiser of East Anglia Quality Assurance meetups where people get together to talk all things ‘testing’.

Ryan believes strongly in making automation testing more accessible to all members of the team and to this end is a strong advocate of ‘Codeless’ automation tools. Ryan also blogs about testing tools and has a strong interest in finding great Python Package to assist in his testing efforts.

Happy Testing!

Join TestProject Community

Get full access to the world's first cloud-based, open source friendly testing community. Enjoy TestProject's end-to-end test automation Platform, Forum, Blog and Docs - All for FREE.

Join Us Now  

Leave a Reply

popup image

The Next Gen of Testing

An exciting TestProject v2 release is coming up! Discover a new generation of testing and get a chance to win PlayStation 5.
Sign me up! right arrow