TestNG is a popular test framework that needs no introduction in the JVM (Java virtual machine) ecosystem and can be used for automating your tests regardless of their size, from large (functional) ones, to medium/small level tests (Integration/Unit tests). It is a highly customizable framework and provides great flexibility around how you want to structure your tests and run them.
Once you have your test suite ready, the next logical step typically entails figuring out and implementing a sensible logging and reporting method to allow you to analyze your executions and possible failures. Though there are multiple frameworks available to implement logging and reporting, in this post we will learn what comes out of the box with TestNG!
Let’s start with a simple example for TestNG reporting and logging
I’ve set up an example demonstrating these TestNG logging and reporting features:
// Dummy set of classes representing an employee where Credited Salary
// gives out the amount to credit based on the employees level
class Level(val employeesLevel: Int)
class Salary(val money: Int)
class Employee(val name: String, val level: Level)
class CreditedSalary(private val level: Level, private val baseSalary: Salary) {
fun getSalaryToCredit(): Int {
return level.employeesLevel * baseSalary.money * 22
}
}
@Test(groups = ["logging_tests"])
class LoggingTests {
private val baseSalary = Salary(1000)
private lateinit var employee: Employee
@BeforeMethod
fun given_EmployeeExists() {
employee = Employee("Rob", Level(2))
}
// Example test that passes
fun when_CreditedSalary_ShouldBeGreater_ThanBase() {
Logger.log("Checking that credited salary is ok")
val credit = CreditedSalary(employee.level, baseSalary)
Assert.assertTrue(credit.getSalaryToCredit() > baseSalary.money)
}
// Example test that fails
fun when_salaryIsNotCredited() {
Logger.log("Checking that salary is not credited")
Assert.assertTrue(false)
}
}
[Line 3 – 13] Employee represents a simple employee in an organization, having a name and level (an integer type value) as its own class.
CreditedSalary is a class that takes a base salary amount and computes the monthly salary to be credited based on the level.
[Line 15 – 36] LoggingTests is a demo class that initializes an employee with a level and has a set of tests.
- when_CreditedSalary_ShouldBeGreater_ThanBase is a test that passes and asserts credited salary is always greater than the base salary.
- when_salaryIsNotCredited is a test that fails.
- Both of these tests belong to logging_tests group.
To allow these tests to be run via Gradle, we can do a base setup with TestNG and a Gradle task that supports running these tests by group name:
...
tasks.withType(Test) {
systemProperties = [
tag: System.getProperty('tag', 'NONE')
]
}
task runTests(type : Test) {
useTestNG() {
testLogging.showStandardStreams = true
includeGroups System.getProperty('tag', 'NONE')
}
}
dependencies {
compile group: 'org.testng', name: 'testng', version: '7.1.0'
...
Run these tests via Gradle
Execute the command below:
./gradlew clean runTests -Dtag=logging_tests
As expected, we can see that Gradle runs our tests and 1 failed:
2 tests completed, 1 failed FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runTests'. > There were failing tests. See the report at: file:///<base_path_till_project>/build/reports/tests/runTests/index.html
Without any special harness/setup, TestNG and Gradle generate a set of reports under path:
file:///<base_path_till_project>/build/reports/tests/runTests/index.html
If we open this site, then we can see our test results with a bunch of useful stats for our run, as seen below:
Also, we can open a failed test to see details with the failure stack trace, as seen below:
Great! 😊 These reports are already quite useful and give us a basic set of insights into our test runs. Let’s extend this a bit further and see what else we can do using TestNG features.
Reporter
Did you notice the statement Logger.log(“”) in our Test class?
TestNG provides a simple set of static logging methods inside Reporter class which allows you to print helpful comments when certain flows/assertions happen during our tests. In our example, we are using it to give an indication of when certain tests happen.
Also, its always a good idea to wrap up such dependencies into our own wrapper in order to give us the flexibility to later change implementation if required without touching multiple files:
object Logger {
fun log(msg: String, stdOut: Boolean = true) {
Reporter.log(msg, stdOut)
}
}
Tip: To ensure TestNG prints these log methods on the terminal (also called Standard Output), ensure you pass the flag as true in the log method. In our utility, we have given it a default value of true.
If we run the test again, we can see that the loggers that we added in our test are printed on the console:
> Task :runTests FAILED
Gradle suite > Gradle test > testFrameworks.testNG.logging.LoggingTests.when_CreditedSalary_ShouldBeGreater_ThanBase STANDARD_OUT
Checking that credited salary is ok
Gradle suite > Gradle test > testFrameworks.testNG.logging.LoggingTests.when_salaryIsNotCredited STANDARD_OUT
Checking that salary is not credited
Gradle suite > Gradle test > testFrameworks.testNG.logging.LoggingTests.when_salaryIsNotCredited FAILED
java.lang.AssertionError at Logging.kt:84
2 tests completed, 1 failed
FAILURE: Build failed with an exception.
Cool 🤩 Also, if you select Standard output in the Gradle results, you can observe the standard output for the two tests is displayed, as seen below:
Where does Gradle and TestNG store this info? You can see the detailed results stored in the below path:
<base_path_till_project>/build/test-results/runTests/TEST-testFrameworks.testNG.logging.LoggingTests.xml
We have all the *.xml files with a detailed stack trace and the standard output messages under <system-out>
<system-out> <![CDATA[Checking that credited salary is ok Checking that salary is not credited ]]> </system-out> <system-err> <![CDATA[]]> </system-err>
It’s important to note that TestNG also exposes an interface called IReporter which has a handle to all the test results once the execution is completed.
Listeners
Apart from the existing Reporter class and the associated interface, TestNG also exposes powerful hooks into its test execution lifecycle by means of an interface called ITestListener
You can either extend this interface and implement all the methods or alternatively extend TestListenerAdapter class which implements ITestListener with empty methods so that you can choose to only override the methods that you care about.
So what if we want to print the name of the test method before and after the test execution to allows us to visually analyze the standard output logs? Let’s implement this in a Listener class which extends our TestListenerAdapter class:
class Listener : TestListenerAdapter() {
override fun onStart(context: ITestContext) {
Logger.log("Beginning test suite: ${context.outputDirectory}")
}
override fun onFinish(context: ITestContext?) {}
override fun onTestSkipped(result: ITestResult?) {}
override fun onTestStart(result: ITestResult) {
super.onTestStart(result)
Logger.log("===>>> Test started: ${result.name}")
}
override fun onTestSuccess(result: ITestResult?) {
super.onTestSuccess(result)
if (result != null) {
Logger.log("<<<=== Test completed successfully: ${result.name}")
}
}
override fun onTestFailure(result: ITestResult) {
super.onTestFailure(result)
Logger.log("<<<=== Test failed: ${result.name}")
}
override fun onTestFailedButWithinSuccessPercentage(result: ITestResult?) {}
}
Here I have still left the overridden methods to show the available options, however, it is not strictly required with the use of TestListenerAdapter and you can choose to implement only the ones that you care about.
So as a use case, what if on every test start and failure, we want to output the name of the method to allow us to figure out the standard logs for that?
If you notice, most of these methods (onTestSkipped, onTestSuccess, onTestFailure, onTestFailedButWithinSuccessPercentage, onTestStart) have an interface called ITestResult as a parameter.
We can get a lot of information about the test run with this and then decide to use this in whichever manner is useful to us. In our example, we have added a logger statement to print the name of the test method in onTestStart and onTestFailure method which would be called before every test execution and whenever a test fails.
Just to give you an idea, below is an example of the set of methods exposed by this interface:
To make it all work, we need to add this as a listener to TestNG. There are a couple of ways to do it, but the most convenient way is to use it at a Gradle or Maven level. Below, I have added the reference path of the class within the useTestNG() method:
task runTests(type : Test) {
useTestNG() {
testLogging.showStandardStreams = true
listeners << "testFrameworks.testNG.logging.Listener"
includeGroups System.getProperty('tag', 'NONE')
}
}
We are all set now💪 Let’s run the tests again and see the Gradle report, we can see our test methods name printed in the standard output and this could help us establish a timeline in the logs and know what events happened leading up to our test’s fail:
Closing Thoughts
I hope this post gives you some insight into the TestNG reporting and logging capabilities you can get out of the box. The power is there 💥 It is up to you to tailor it to your needs. If you found this useful, go ahead and share it with your friends and colleagues.
Here are some more references for you to meditate on this further:
Related posts on TestNG features: