HTML Test Reports for Selenium & Appium Python Test Automation

As you might have read in the previous blog posts on the brand new TestProject Python SDK (Part 1 and Part 2), the SDK will generate great-looking HTML test reports and automatically publish them on the TestProject platform for you, out of the box, no additional configuration needed (you can also download them as a PDF file). But did you know that the reporting is also highly configurable? In this blog post, you’ll see all the options you have with regards to test reports and how to customize them according to your requirements in just a couple of short minutes.

🐍 Watch this awesome webinar recording to get started with the TestProject Python OpenSDK! 🐍

Getting Started with TestProject Python SDK
Creating Mobile Appium Tests with TestProject Python SDK
You’re here →HTML Test Reports for Selenium & Appium Python Test Automation

Test reports are crucial in any test automation framework, and we all know how hard even an expert needs to work in order to generate test automation reports, not to mention if we want to have customization freedom too… There are many tailored solutions available in the industry to help us overcome this, such as: JUnit, Jenkins plugin, ELK, Grafana, or even developing customized web dashboards.

However, none are out of the box test reporting solutions. Enter TestProject, which provides users with a complete dedicated reporting environment that supports Selenium, Appium and Python, all for free. TestProject automatically creates the HTML and PDF reports for you (out of the box, no additional configuration needed).

All you need to do is sign up for a free TestProject account, download and install the TestProject Agent (that already contains all the Selenium/Appium drivers for you), install the TestProject Python SDK using pip (pip install testproject-python-sdk), and configure your developer token. That’s it! (For more info, follow this Github link, or read through this great getting started tutorial).

With TestProject you get free access to extensive reporting capabilities which can be fully customizable (as will be described in this article below), including: Detailed analytics dashboards in the cloud, screenshots, pass/fail criteria, custom error messages, ability to download reports to PDF, easily share reports with teammates, RESTful API access, and 100% Selenium and Appium compatibility (and not only with Python, but with Java and C# too!).

Out of the box HTML test reports

If you leave all settings as they are, the SDK will:

Generate reports with the project, job and test names that are automatically inferred (see for more details this article).
Automatically report a new test when the quit() command is called on a driver, or when the name of the test method that is executed changes.
Contain all WebDriver commands that were executed, along with their result (pass or fail).
Redact values that are typed in sensitive (passwords) text field elements.

All of these options are configurable, as you will see soon enough.

Specify custom project and job names

If you want to override the automatically inferred project and job names that are displayed in your TestProject reports, you can do that in two ways. The first way is by passing custom project and job names as arguments to the driver constructor:

from src.testproject.sdk.drivers import webdriver

def test_custom_project_and_job_names():
    driver = webdriver.Chrome(projectname="My project", jobname="Reporting job")
    driver.get("https://example.testproject.io/web/")

The second way is by specifying them in the @report decorator:

from src.testproject.decorator import report
from src.testproject.sdk.drivers import webdriver

@report(project="My project", job="Reporting job")
def test_project_and_job_names_in_decorator():
    driver = webdriver.Chrome()
    driver.get("https://example.testproject.io/web/")

Both options will result in the specified project and job names being displayed in TestProject:

Specifying a custom test name

If you want a custom test name in your report (i.e., not the name of the test method), you can use the @report decorator for that as well:

from src.testproject.decorator import report
from src.testproject.sdk.drivers import webdriver

@report(test="My custom test name")
def test_name_in_decorator():
    driver = webdriver.Chrome()
    driver.get("https://example.testproject.io/web/")

When specified like this, the custom test name will be used instead of the automatically inferred one:

Manual test and step reporting

By default, the SDK automatically reports a new test when the quit() command is called on a driver, or when the name of the test method that is executed changes. This behaviour can be disabled:

from src.testproject.sdk.drivers import webdriver

def test_disable_automatic_test_reporting():
    driver = webdriver.Chrome()
    driver.report().disable_auto_test_reports(disabled=True)
    driver.get("https://example.testproject.io/web/")

You can instead report a test manually like this:

from src.testproject.sdk.drivers import webdriver

def test_report_a_test_manually():
    driver = webdriver.Chrome()
    driver.report().disable_auto_test_reports(disabled=True)
    driver.get("https://example.testproject.io/web/")
    driver.report().test(name="A manually reported test", passed=True)

This will result in the following test being reported on the TestProject platform:

Additionally, you can also report intermediate steps manually, and even add screenshots when you do so:

from src.testproject.sdk.drivers import webdriver

def test_report_a_step_manually():
    driver = webdriver.Chrome()
    driver.report().disable_auto_test_reports(disabled=True)
    driver.get("https://example.testproject.io/web/")
    driver.report().step(description="An intermediate test step", message="An additional message", passed=False, screenshot=True)
    driver.report().test(name="A manually reported test", passed=True)

which will lead to this step being included in the test report:

As you can see, the step includes a screenshot (that will be enlarged when clicked). Also note that even though we set the manually reported test to passed in our code, it is marked as failed because of the failed step that is part of the test.

Disable redaction of text typed into secured text fields

By default, the SDK redacts text that is typed into a text field that is deemed to contain sensitive data, i.e.:

Fields that have a type attribute with value password (all browser and OS types).
Fields of type XCUIElementTypeSecureTextField (only on iOS with XCUITest).

Text typed in such a text field is replaced by asterisks in the report:

If you wish, this can be disabled:

def test_disable_command_redaction():
    driver = webdriver.Chrome()
    driver.report().disable_auto_test_reports(disabled=True)
    driver.get("https://example.testproject.io/web/")
    driver.report().disable_redaction(disabled=True)
    driver.find_element_by_css_selector("#name").send_keys("John Smith")
    driver.find_element_by_css_selector("#password").send_keys("12345")

which leads to password values being reported in plain text, instead:

Disabling reporting

Finally, you also can disable various reporting to various degrees.

If you don’t want to report WebDriver commands automatically, you can specify that as follows:

def test_disable_driver_command_reporting():
    driver = webdriver.Chrome()
    driver.report().disable_command_reports(disabled=True)
    driver.get("https://example.testproject.io/web/")  # this command will not be reported
    driver.report().disable_command_reports(disabled=False)
    driver.find_element_by_css_selector("#name").send_keys("John Smith")  # this command will be reported

If you want to disable all reporting temporarily, you can do that, too:

def test_disable_all_reporting_temporarily():
    driver = webdriver.Chrome()
    driver.report().disable_reports(disabled=True)
    # nothing will be reported here
    driver.report().disable_reports(disabled=False)

And if you want to permanently disable reporting for a specific driver session, you can do that, too:

def test_project_and_job_names_in_decorator():
    driver = webdriver.Chrome(disable_reports=True)

Please note that in the last case, you cannot re-enable reporting for the duration of the driver session!

As you can see, the TestProject Python SDK offers a variety of options to create the reporting you want very easily, solving one of the greatest challenges in open source testing. From now on, test reporting is a breeze thanks to the TestProject new OpenSDK.

Happy PyTesting! 🐍