logo logo

Non Blocking Assertion Failures with Pytest-check

Pytest Check

There is much debate in the testing community about how many assertions a single UI automated test should have. Some standby the one test, one assert method, meaning that each test should only check or validate a single item. Others are quite happy for the test to check multiple items throughout.

Whatever the approach, I think it’s pretty safe to say that tests should remain clear, concise, readable and of course maintainable. I personally have no issue with multiple assertions in a test, so long as I’m only focusing on one area of functionality.

Take this Registration form, for instance:

Registration Form

As part of a Register User test, I may want to check a couple of things. Perhaps check that the congratulations message popup is displayed and maybe that the user is routed to the login page following successful registration.

I wouldn’t then want to test that the user can login successfully, I can do that part in another test.

The Problem

The trouble with Python’s Inbuilt assert method is that it stops the test on first failure. This makes perfect sense in some scenarios. There is no point in continuing a test if part of the application is not working. However, there are cases where it makes more sense to make all your checks regardless of if one fails, especially if all your assertions relate to a single page at the end of your test.

This is an approach we often use in web service testing. When validating a response body, we probably want to check each value returned and then report on which ones were not as expected rather than stopping the test when the first assert fails.

The Solution: Pytest-check Plugin

Pytest-check (created by Brian Okken) is a Pytest plugin that enables you to wrap up all test assertions into a single pass/fail result. For example, if you have 3 assertions in a test and the first one fails, Pytest-check will continue to run the remaining 2 assertions. It will then report the test as failed if one or more checks fail. Nice, huh? ✨

❗ TestProject’s Python OpenSDK also supports Pytest – So if you have existing pytest-based Selenium tests, it is very easy to convert these to TestProject-powered tests. You can read more about it in my latest article over at HowQA Blog, and check out a step-by-step getting started tutorial here.

Let’s see how the Pytest-check plugin works 👇

Selenium Test

First, we need a test. I’ll be using the registration page of this test application: https://docket-test.herokuapp.com/register

import selenium.webdriver as webdriver
from selenium.webdriver.common.by import By


def test_register_user():
    # Arrange
    url = "https://docket-test.herokuapp.com/register"
    # set the driver instance
    driver = webdriver.Chrome()
    # browse to the endpoint
    driver.get(url)
    # maximise the window
    driver.maximize_window()

    # Act
    # Complete registration form

    # enter username value
    driver.find_element(By.ID, "username").send_keys("Ryan")
    # enter email value
    driver.find_element(By.ID, "email").send_keys("[email protected]")
    # enter password value
    driver.find_element(By.ID, "password").send_keys("12345")
    # enter repeat password value
    driver.find_element(By.ID, "password2").send_keys("12345")
    # click register button
    driver.find_element(By.ID, "submit").click()

We have a test, but we need to write our checks. First let’s do that using assert method:

# Assert
# confirm registration has been successful

# check if congratulations message contains the correct text
message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
assert message == "Congratulations, you are now registered"

# check user is routed to login page
current_url = driver.current_url
assert current_url == "https://docket-test.herokuapp.com/login"

If we run it now, everything should pass:

Selenium Test using Assert method

Great so far, but what happens if both our assertions fail? Let’s change our assertions so we can see:

# Assert
# confirm registration has been successful

# check if congratulations message contains the correct text
message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
assert message == "Well done, You've Registered"

# check user is routed to login page
current_url = driver.current_url
assert current_url == "https://docket-test.herokuapp.com/register"

driver.quit()

So, we’ve updated our expected message and changed the expected URL so if we run this test again it will now fail:

Selenium Test

As expected, the test fails with an assertion error.
Great! Our test has done its job, we’ve found a bug, well, sort of 🤔

What you can see from the error message though is that the test has failed due to the wording in the message. It hasn’t even checked the URL. The test stops as soon as it finds the first failure, so we have no way of telling from the result if that was correct, hmmm… In this scenario, we would have to fix our expected message text and then rerun the test. So let’s do that now.

Now we’ve changed the expected message back to ‘Congratulations, you are now registered’ now we can run it again:

Selenium Test

Ah ha! Another failure, this time due to the expected URL.

Yeah, I know what you’re thinking wouldn’t it be great if we could have just captured both of these failures in one test run, well you’re in luck. Enter Pytest-check.

Pytest-Check

Installing

We can install pytest-check using Pip install pytest-check Once installed pytest-check we can import it into our test with.

import pytest_check as check

Ok, now that’s done, we can make some changes to our assertions. From now on we’re not going to be using the assert statement, instead, we’ll use the pytest-check syntax as follows.

To check the message we’ll use the check.equal function and provide the Expected and the actual text, like so:

check.equal(message, "Congratulations, you are now registered1")

We can do the same thing for our URL check but for the purposes of showcasing a different method, we can use check.is_in instead.

check.is_in("login", current_url)

Our full test now looks like this:

import selenium.webdriver as webdriver
from selenium.webdriver.common.by import By
import pytest_check as check


def test_register_user():
    # Arrange
    url = "https://docket-test.herokuapp.com/register"
    # set the driver instance
    driver = webdriver.Chrome()
    # browse to the endpoint
    driver.get(url)
    # maximise the window
    driver.maximize_window()

    # Act
    # Complete registration form

    # enter username value
    driver.find_element(By.ID, "username").send_keys("Ryan8")
    # enter email value
    driver.find_element(By.ID, "email").send_keys("[email protected]")
    # enter password value
    driver.find_element(By.ID, "password").send_keys("12345")
    # enter repeat password value
    driver.find_element(By.ID, "password2").send_keys("12345")
    # click register button
    driver.find_element(By.ID, "submit").click()

    # Assert
    # confirm registration has been successful

    # check if congratulations message contains the correct text
    message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
    check.equal(message, "Congratulations, you are now registered")

    # check user is routed to login page
    current_url = driver.current_url
    check.is_in("login", current_url)

    driver.quit()

At this stage We can also revert the expected values back to their original state so the test will pass if we run it. So let’s do that first.

Pytest-check plugin - Selenium Test

Great! Now let’s see what happens if both our checks fail. First, we’ll need to update the code to make the checks fail once again:

# check if congratulations message contains the correct text
message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
check.equal(message, "Congratulations, you are now registered!")

# check user is routed to login page
current_url = driver.current_url
check.is_in("1", current_url)

So let’s run it.

Pytest-check plugin - Selenium Test

Like last time our test has failed as expected, but this time it’s recorded that 2 checks failed: first it reports that the message is not as expected, followed by the URL check. As far as pytest is concerned this is treated as a single test failure but we now know that multiple checks have failed.

Only if all the checks in the test pass, will the test be marked as a pass ✅

Awesome, well that’s Pytest-check. Find out more by reading the documentation here.

Happy Testing Everyone! 🐍😉

About the author

Ryan Howard

Ryan is a Test Engineer and founder of How QA Test consultancy with 10 years experience in the software industry. Ryan loves to talk testing and is the co organiser of East Anglia Quality Assurance meetups where people get together to talk all things ‘testing’.

Ryan believes strongly in making automation testing more accessible to all members of the team and to this end is a strong advocate of ‘Codeless’ automation tools. Ryan also blogs about testing tools and has a strong interest in finding great Python Package to assist in his testing efforts.

Happy Testing!

Leave a Reply

FacebookLinkedInTwitterEmail