logo logo

TestNG Reporting and Logging Built-in Features

main post image

TestNG is a popular test framework that needs no introduction in the JVM (Java virtual machine) ecosystem and can be used for automating your tests regardless of their size, from large (functional) ones, to medium/small level tests (Integration/Unit tests). It is a highly customizable framework and provides great flexibility around how you want to structure your tests and run them.

Once you have your test suite ready, the next logical step typically entails figuring out and implementing a sensible logging and reporting method to allow you to analyze your executions and possible failures. Though there are multiple frameworks available to implement logging and reporting, in this post we will learn what comes out of the box with TestNG!

Let’s start with a simple example for TestNG reporting and logging

I’ve set up an example demonstrating these TestNG logging and reporting features:

// Dummy set  of classes representing an employee where Credited Salary
// gives out the amount to credit based on the employees level
class Level(val employeesLevel: Int)

class Salary(val money: Int)

class Employee(val name: String, val level: Level)

class CreditedSalary(private val level: Level, private val baseSalary: Salary) {
    fun getSalaryToCredit(): Int {
        return level.employeesLevel * baseSalary.money * 22
    }
}

@Test(groups = ["logging_tests"])
class LoggingTests {
    private val baseSalary = Salary(1000)
    private lateinit var employee: Employee

    @BeforeMethod
    fun given_EmployeeExists() {
        employee = Employee("Rob", Level(2))
    }

    // Example test that passes
    fun when_CreditedSalary_ShouldBeGreater_ThanBase() {
        Logger.log("Checking that credited salary is ok")
        val credit = CreditedSalary(employee.level, baseSalary)
        Assert.assertTrue(credit.getSalaryToCredit() > baseSalary.money)
    }

    // Example test that fails
    fun when_salaryIsNotCredited() {
        Logger.log("Checking that salary is not credited")
        Assert.assertTrue(false)
    }
}

[Line 3 – 13] Employee represents a simple employee in an organization, having a name and level (an integer type value) as its own class.
CreditedSalary is a class that takes a base salary amount and computes the monthly salary to be credited based on the level.

[Line 15 – 36] LoggingTests is a demo class that initializes an employee with a level and has a set of tests.

  • when_CreditedSalary_ShouldBeGreater_ThanBase is a test that passes and asserts credited salary is always greater than the base salary.
  • when_salaryIsNotCredited is a test that fails.
  • Both of these tests belong to logging_tests group.

To allow these tests to be run via Gradle, we can do a base setup with TestNG and a Gradle task that supports running these tests by group name:

...

tasks.withType(Test) {
    systemProperties = [
            tag: System.getProperty('tag', 'NONE')
    ]
}

task runTests(type : Test) {
    useTestNG() {
        testLogging.showStandardStreams = true
        includeGroups System.getProperty('tag', 'NONE')
    }
}

dependencies {
    compile group: 'org.testng', name: 'testng', version: '7.1.0'
    
    ...

Run these tests via Gradle

Execute the command below:

./gradlew clean runTests -Dtag=logging_tests

As expected, we can see that Gradle runs our tests and 1 failed:

2 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':runTests'.
> There were failing tests. See the report at: file:///<base_path_till_project>/build/reports/tests/runTests/index.html

Without any special harness/setup, TestNG and Gradle generate a set of reports under path:

file:///<base_path_till_project>/build/reports/tests/runTests/index.html

If we open this site, then we can see our test results with a bunch of useful stats for our run, as seen below:

Gradle TestNG Report

Also, we can open a failed test to see details with the failure stack trace, as seen below:

Gradle TestNG Report

Great! 😊 These reports are already quite useful and give us a basic set of insights into our test runs. Let’s extend this a bit further and see what else we can do using TestNG features.

Reporter

Did you notice the statement Logger.log(“”) in our Test class?

TestNG provides a simple set of static logging methods inside Reporter class which allows you to print helpful comments when certain flows/assertions happen during our tests. In our example, we are using it to give an indication of when certain tests happen.

Also, its always a good idea to wrap up such dependencies into our own wrapper in order to give us the flexibility to later change implementation if required without touching multiple files:

object Logger {
    fun log(msg: String, stdOut: Boolean = true) {
        Reporter.log(msg, stdOut)
    }
}

Tip: To ensure TestNG prints these log methods on the terminal (also called Standard Output), ensure you pass the flag as true in the log method. In our utility, we have given it a default value of true.

If we run the test again, we can see that the loggers that we added in our test are printed on the console:

> Task :runTests FAILED

Gradle suite > Gradle test > testFrameworks.testNG.logging.LoggingTests.when_CreditedSalary_ShouldBeGreater_ThanBase STANDARD_OUT
    Checking that credited salary is ok

Gradle suite > Gradle test > testFrameworks.testNG.logging.LoggingTests.when_salaryIsNotCredited STANDARD_OUT
    Checking that salary is not credited

Gradle suite > Gradle test > testFrameworks.testNG.logging.LoggingTests.when_salaryIsNotCredited FAILED
    java.lang.AssertionError at Logging.kt:84

2 tests completed, 1 failed

FAILURE: Build failed with an exception.

Cool đŸ€© Also, if you select Standard output in the Gradle results, you can observe the standard output for the two tests is displayed, as seen below:

Gradle TestNG Reporting

Where does Gradle and TestNG store this info? You can see the detailed results stored in the below path:

<base_path_till_project>/build/test-results/runTests/TEST-testFrameworks.testNG.logging.LoggingTests.xml

We have all the *.xml files with a detailed stack trace and the standard output messages under <system-out>

<system-out>
  <![CDATA[Checking that credited salary is ok Checking that salary is not credited ]]>
</system-out> 
<system-err>
  <![CDATA[]]>
</system-err>

It’s important to note that TestNG also exposes an interface called IReporter which has a handle to all the test results once the execution is completed.

Listeners

Apart from the existing Reporter class and the associated interface, TestNG also exposes powerful hooks into its test execution lifecycle by means of an interface called ITestListener

You can either extend this interface and implement all the methods or alternatively extend TestListenerAdapter class which implements ITestListener with empty methods so that you can choose to only override the methods that you care about.

So what if we want to print the name of the test method before and after the test execution to allows us to visually analyze the standard output logs? Let’s implement this in a Listener class which extends our TestListenerAdapter class:

class Listener : TestListenerAdapter() {
    override fun onStart(context: ITestContext) {
        Logger.log("Beginning test suite: ${context.outputDirectory}")
    }

    override fun onFinish(context: ITestContext?) {}
    override fun onTestSkipped(result: ITestResult?) {}

    override fun onTestStart(result: ITestResult) {
        super.onTestStart(result)
        Logger.log("===>>> Test started: ${result.name}")
    }

    override fun onTestSuccess(result: ITestResult?) {
        super.onTestSuccess(result)
        if (result != null) {
            Logger.log("<<<=== Test completed successfully: ${result.name}")
        }
    }

    override fun onTestFailure(result: ITestResult) {
        super.onTestFailure(result)
        Logger.log("<<<=== Test failed: ${result.name}")
    }

    override fun onTestFailedButWithinSuccessPercentage(result: ITestResult?) {}
}

Here I have still left the overridden methods to show the available options, however, it is not strictly required with the use of TestListenerAdapter and you can choose to implement only the ones that you care about.

So as a use case, what if on every test start and failure, we want to output the name of the method to allow us to figure out the standard logs for that?

If you notice, most of these methods (onTestSkipped, onTestSuccess, onTestFailure, onTestFailedButWithinSuccessPercentage, onTestStart) have an interface called ITestResult as a parameter.

We can get a lot of information about the test run with this and then decide to use this in whichever manner is useful to us. In our example, we have added a  logger statement to print the name of the test method in onTestStart and onTestFailure method which would be called before every test execution and whenever a test fails.

Just to give you an idea, below is an example of the set of methods exposed by this interface:

TestNG Reporting

To make it all work, we need to add this as a listener to TestNG. There are a couple of ways to do it, but the most convenient way is to use it at a Gradle or Maven level. Below, I have added the reference path of the class within the useTestNG() method:

task runTests(type : Test) {
    useTestNG() {
        testLogging.showStandardStreams = true
        listeners << "testFrameworks.testNG.logging.Listener"
        includeGroups System.getProperty('tag', 'NONE')
    }
}

We are all set nowđŸ’Ș Let’s run the tests again and see the Gradle report, we can see our test methods name printed in the standard output and this could help us establish a timeline in the logs and know what events happened leading up to our test’s fail:

TestNG Reporting

Closing Thoughts

I hope this post gives you some insight into the TestNG reporting and logging capabilities you can get out of the box. The power is there đŸ’„ It is up to you to tailor it to your needs. If you found this useful, go ahead and share it with your friends and colleagues.

Here are some more references for you to meditate on this further:

Related posts on TestNG features:

Avatar

About the author

Gaurav Singh

I have a strong passion for Test engineering and automation and love to code in Kotlin, Java, and Python to build scalable test automation frameworks and utilities that solve hard testing problems. My Day job is at Gojek Bangalore as a QA Lead.

I have experience in Web and Mobile UI automation using Selenium and Appium and API automation using RestAssured and python and have a fair understanding of Docker and GitLab to setup CI pipelines. On the “ilities” side, I have wired load tests for API’s using Locust framework

I also blog about my journey as a tester at https://automationhacks.blog/ and am an amateur conference speaker and like to conduct tech talks and knowledge sharing sessions.

Join TestProject Community

Get full access to the world's first cloud-based, open source friendly testing community. Enjoy TestProject's end-to-end test automation Platform, Forum, Blog and Docs - All for FREE.

Join Us Now  

Leave a Reply

popup image

Best In Class Java and C# SDK

Join thousands of automation developers using TestProject to supercharge open source testing, with a Selenium and Appium SDK, supporting Java and .NET Core (C#)!
Sign Up Now right arrow
FacebookLinkedInTwitterEmail