Selenium is a great tool, without a doubt. The real value of tests written in Selenium can be harvested only based on the test results it publishes. We have ‘n’ number of reporting tools & frameworks for Selenium as such. But the common underlying issues with any of these reporting tools are they are mostly OFFLINE, which means the test results are published only after the test execution.
Let’s assume a business case for this solution. We have 1000 automated test cases getting executed in the build pipeline. The status of the tests is published only after the test execution completes. What if there is a series of tests failing among the 1000 tests because of the same issue in the AUT? As it is probably a headless or remote execution, we will not be able to see it and to the extent, we will need to see build logs. And we all know monitoring logs manually is a big headache for testers! 🤕
Welcome to the World of Test Monitoring!
In this article, we are going to learn about how to get the results of our Selenium tests LIVE using Grafana and InfluxDB. Let’s get started! 📊👉
❗ Please note:
- This article is written for macOS. So please refer to Grafana and InfluxDB sites for the Windows version of the installation.
- There are of course additional open source and free platforms such as TestProject that have built-in detailed reports and dashboards for your Selenium and Appium tests. You can execute tests and get your reports right away, get Slack and email notifications, download full reports, and also easily integrate this flow within your build pipeline utilizing their extensive API.
- References:
Test Monitoring – Make your Selenium Test Results Live using Grafana & InfluxDB
- What is InfluxDB
- What is Grafana
- InfluxDB Installation & Configuration
- Grafana Installation & Configuration
- Using InfluxDB in Grafana
- Configure Charts in Grafana
- Conclusion
What is InfluxDB?
InfluxDB is an open-source time-series database (TSDB) developed by InfluxData.
What is Grafana?
Grafana is an open-source metric analytics & visualization suite.
InfluxDB Installation & Configuration (macOS)
Run the below command to install InfluxDB via brew:
brew install influxdb
Once installed, now run the below command to start the InfluxDB server:
brew services start influxdb
❗ Tip: To Stop Influxdb, run this command:
brew services stop influxdb
After the service is started, let’s create a Database. To do this, you need to run influxd first. So, type:
influxd
Now, type the query to create a database:
create database selenium_test_results;
So, the overall steps should look something like this:
That’s it, we have successfully installed and configured InfluxDB.
❗ Note: Do NOT worry about any table creation. We will handle this through our Java code.
Grafana Installation & Configuration (Mac OS)
We will again use brew to install Grafana as it is very easy. Run the below command:
brew install grafana
Once installed, let’s start it. To start the instance:
brew services start grafana
❗ Tips: To Stop Grafana, run this command:
brew services stop grafana
Once all done, you should see the terminal like this –
Now, let’s launch the Grafana Dashboard. Go to your favorite browser and launch this URL: http://localhost:3000
❗ Note: Username: admin & Password: admin
Using InfluxDB in Grafana
To use InfluxDB in Grafana, we need to establish a Data Source Connection.
Go to Configuration –> Data Sources –> Add Data Source –> InfluxDB
❗ Note: Username and Password – root
Once configured, check whether the Data Source works by clicking the Save & Test button.
Java Code
We will create a Maven Project for the purpose of this article.
Add influxdb-java dependency in your pom.xml
<dependency> <groupId>org.influxdb</groupId> <artifactId>influxdb-java</artifactId> <version>2.12</version> </dependency>
Great! Let’s create the Listener Class now. Create a new class called InfluxDBListener and copy & paste the below code there:
InfluxDBListener.java
package com.utils;
import org.influxdb.dto.Point;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;
import java.util.concurrent.TimeUnit;
public class InfluxDBListener implements ITestListener {
public void onTestStart(ITestResult iTestResult) {
}
public void onTestSuccess(ITestResult iTestResult) {
this.postTestMethodStatus(iTestResult, "PASS");
}
public void onTestFailure(ITestResult iTestResult) {
this.postTestMethodStatus(iTestResult, "FAIL");
}
public void onTestSkipped(ITestResult iTestResult) {
this.postTestMethodStatus(iTestResult, "SKIPPED");
}
public void onTestFailedButWithinSuccessPercentage(ITestResult iTestResult) {
}
public void onStart(ITestContext iTestContext) {
}
public void onFinish(ITestContext iTestContext) {
this.postTestClassStatus(iTestContext);
}
private void postTestMethodStatus(ITestResult iTestResult, String status) {
Point point = Point.measurement("testmethod").time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
.tag("testclass", iTestResult.getTestClass().getName()).tag("name", iTestResult.getName())
.tag("description", iTestResult.getMethod().getDescription()).tag("result", status)
.addField("duration", (iTestResult.getEndMillis() - iTestResult.getStartMillis())).build();
UpdateResults.post(point);
}
private void postTestClassStatus(ITestContext iTestContext) {
Point point = Point.measurement("testclass").time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
.tag("name", iTestContext.getAllTestMethods()[0].getTestClass().getName())
.addField("duration", (iTestContext.getEndDate().getTime() - iTestContext.getStartDate().getTime()))
.build();
UpdateResults.post(point);
}
}
To update the results to the InfluxDB, we need to create a class for connection like below:
UpdatesResults.java
package com.utils;
import org.influxdb.InfluxDB;
import org.influxdb.InfluxDBFactory;
import org.influxdb.dto.Point;
public class UpdateResults {
private static final InfluxDB INFLXUDB = InfluxDBFactory.connect("http://localhost:8086", "root", "root");
private static final String DB_NAME = "selenium_test_results";
static {
INFLXUDB.setDatabase(DB_NAME);
}
public static void post(final Point point) {
INFLXUDB.write(point);
}
}
Now, let’s create some tests:
SwagLabsTests.java
package com.tests;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.support.ui.Select;
import org.testng.Assert;
import org.testng.annotations.Test;
public class SwagLabsTests {
WebDriver driver;
public static int getRandomInteger(int maximum, int minimum) {
return ((int) (Math.random() * (maximum - minimum))) + minimum;
}
@Test(description = "Launch and Login")
public void launchApp() throws Exception {
try {
driver.get("https://www.saucedemo.com/");
driver.findElement(By.id("user-name")).sendKeys("standard_user");
driver.findElement(By.id("user-name")).sendKeys("secret_sauce");
driver.findElement(By.id("user-name")).submit();
driver.quit();
System.out.println("Launch and Login - Done");
} catch (Exception e) {
e.printStackTrace();
}
}
@Test(description = "Sort the Products", dependsOnMethods = "launchApp")
public void sort() throws Exception {
try {
driver.get("https://www.saucedemo.com/");
driver.findElement(By.id("user-name")).sendKeys("standard_user");
driver.findElement(By.id("user-name")).sendKeys("secret_sauce");
driver.findElement(By.id("user-name")).submit();
Select sort = new Select(driver.findElement(By.className("product_sort_container")));
sort.selectByValue("lohi");
driver.quit();
System.out.println("Sort the Products - Done");
} catch (Exception e) {
e.printStackTrace();
}
}
@Test(description = "Select a Product", dependsOnMethods = "sort")
public void selectAProduct() throws Exception {
try {
driver.get("https://www.saucedemo.com/");
driver.findElement(By.id("user-name")).sendKeys("standard_user");
driver.findElement(By.id("user-name")).sendKeys("secret_sauce");
driver.findElement(By.id("user-name")).submit();
driver.findElement(By.linkText("Sauce Labs Backpack")).click();
Assert.assertEquals(driver.getTitle().toString(), "Swag Labs");
driver.quit();
System.out.println("Select a Product - Done");
} catch (Exception e) {
e.printStackTrace();
}
}
@Test(description = "Go to About Page", dependsOnMethods = "selectAProduct")
public void goToAboutPage() throws Exception {
try {
driver.get("https://www.saucedemo.com/");
driver.findElement(By.id("user-name")).sendKeys("standard_user");
driver.findElement(By.id("user-name")).sendKeys("secret_sauce");
driver.findElement(By.id("user-name")).submit();
driver.findElement(By.xpath("//*[@id=\"menu_button_container\"]/div/div[3]/div/button")).click();
driver.findElement(By.linkText("About")).click();
driver.quit();
System.out.println("Go to About Page - Done");
} catch (Exception e) {
e.printStackTrace();
}
}
}
Create a testng.xml file now by right-clicking the project –> TestNG –> Convert to TestNG
testNG.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="My suite">
<listeners>
<listener class-name="com.utils.InfluxDBListener" />
</listeners>
<test name="Test1">
<classes>
<class name="com.tests.SwagLabsTests"/>
</classes>
</test>
</suite>
Now that we have our tests, listener and testng.xml ready, it’s time for us to execute the tests.
❗ Note: Make sure you run the tests from testng.xml and NOT as Test Class as the InfluxDB listener is required to update the test results.
Configure Charts in Grafana
Now, the interesting part! 🎉 Let’s create two graphs that will show us the test results dynamically.
Test Results Count Pie-Chart
In Grafana, create a new Pie-Chart and configure the query as below.
❗ Tip: In case you don’t have Pie-Chart, follow this link to install the plugin.
Test Case Execution Time Point Graph
Create a Graph and configure the query as below:
Time to execute the tests!
Let’s open the testNG.xml file and run the tests. Navigate to Grafana and see the test results updating dynamically.
To see the test execution, see this video.
Conclusion
This is how we make the Selenium Test Results Live, using Grafana and InfluxDB! This is just one of the many methods out there for test monitoring and reporting, which is a crucial process for our automation testing. Therefore, in my next article, I will be discussing another tool, this time the free test automation platform TestProject, and see how we can leverage their extensive test reports and dashboards without having to use any other external tools, just TestProject.
You can find all the code from this article here.
Happy Reporting! 😎✅