logo logo

The 8 Leading Test Reporting Tools for Selenium

main post image

Selenium is the leading open source solution for web test automation, and according to our recent State of Open Source Testing Survey, Selenium is without a doubt the king of the web with over 80% of worldwide respondents surveyed that have chosen Selenium as their preferred functional testing tool. But as we all know Selenium is only a driver, and when it comes to test reporting, Selenium does not have any built-in features for that.

Test reports are a crucial part of any test automation framework, yet surprisingly enough there are very few dedicated reporting tools for automated testing available out there 😧

In this article, we will review the pros and cons of 8 notable test reporting solutions for generating Selenium automation reports: TestNG, Extent Reports / Klov, Robot Framework, TestProject’s OpenSDK, Allure, JUnit plugin for Jenkins, JUnit and Azure DevOps.

Leading Test Reporting Tools for Selenium

ToolLanguageLicenseProsCons
TestNG
TestNG
JavaOpen source*Part of TestNG automation framework
*Create HTML reports
*Popular among Java developers
*Support only Java
*Tightly coupled with TestNG framework
*Primitive look & feel
*Creates only local HTML reports

Extent Reports / Klov

ExtentReports

Java
C#
Commercial & community edition*On-premise solution
*Has an extensive community plan
*The reports look modern
*Advanced filters and screenshots
*Integration with CI/CD
*Good documentation
*The community plan doesn’t include CI/CD integration
*On-premise solution meaning that you need to maintain it
*Support only Java and C#
*The community plan has a limited dashboard that does not include: Errors able, Tags table, Author filter, Device filter

TestProject OpenSDK

TestProject

Python
Java
C#
Open Source*Creates automatic reports in a Cloud-based environment
*Report customization options
*Modern reports, including screenshots
*Ability to bind with any existing automation framework based on Selenium/Appium
*You can extract reports to PDF, share via link/URL, or extract reports via RESTful API
*Has its own DB with free storage (stand-alone solution)
*Fixed reporting HTML structure
*You need to execute tests with the TestProject Agent, that is responsible for generating the reports
*The reports are stored in the AWS based TestProject cloud. There is no on-premise solution

Robot Framework

Robot Framework

PythonOpen source*Part of Robot automation framework
*Capable to create HTML reports
*Supports KDT reports
*Supports only Python
*You’re forced to work according to KDT approach
*Primitive look & feel
*Creates only local HTML reports

Allure

Allure

Java
PHP
Ruby
Python
Scala
C#
Open source*Has multiple adapters for many testing frameworks
*Modern style reports
*On-premise solution
*Supports screenshots
*Nice community and Git contributors
*A lot of links in documentation and examples are broken
*Plugin solution that does not have its own database, needs to be integrated with Jenkins / TeamCity /Bamboo to turn in to a collaborative report

Jenkins + JUnit plugin

Jenkins

JavaOpen source*You can have your test automation reports in Jenkins
*Can integrate with any test automation framework as long as it generates the reports in JUnit format
*Build-in for Junit/TestNG
*Works only with Jenkins
*You need to make sure to generate your reports in JUnit format
*Mainly fits CI/CD development flows, DevOps configuration work is required to make it work
*Doesn’t provide a solution for local execution
*The reports are quite primitive in look & feel

JUnit
JUnit

 

JavaOpen source*Integrated with JUnit framework
*Popular among Java developers
*Creates local HTML reports
*Works only with Java & JUnit
*Primitive look & feel
*Required code implementation to make it work
*Local reports only

Azure DevOps

Azure DevOps

C#
Java
JavaScript
Python
Commercial product*Automated tests configured to run as part of a build
*Cloud-based centralized reports
*Supports multiple automation frameworks
*Summary and detailed reports
*Can include screenshots and videos
*Doesn’t fit non-Azure DevOps users, the reporting feature is tightly coupled with Microsoft cloud echo system
*Reports are built as part of CI/CD process, so pipelines need to be configured, thus will take time to setup and configure
*If you use Jenkins/TeamCity/Travis/Bamboo/Github or other CI, this solution is not for you.
*If you don’t use Visual Studio as your IDE it will be less convenient or almost impossible to use
*If you need an on-premise solution you can’t use Azure DevOps


Conclusion 

When it comes to collaborative automated testing reports, Extent Reports (Klov) and TestProject are the only neutral solutions we found as of today that have their test automation results own repositories. Here are the main difference between the two (you can explore both of their docs here: Klov documentationTestProject documentation):

  • Klov is on-premise and needs setup, whereas TestProject is a cloud-based tool so you just create an account.
  • Klov has a commercial paid plan and a limited community addition, whereas TestProject is a fully-featured community tool with open source SDKs and it is completely free.
  • Klov supports Java & C#, whereas TestProject supports Python, Java and C#.
  • Klov’s API has assertions to create the reports, whereas TestProject creates reports automatically based on Selenium calls and also has the ability to create custom reports based on assertions.

If you are going for an on-premise solution, Allure is worth trying too, but it feels a bit pre-mature (some documentation is missing) and you should have a CI/CD tool and some DevOps resources to set it up and make it work.

If you are familiar with additional test automation test reporting tools for Selenium, please share in the comments and we will be happy to add them to our list! 😉

Meital Matsafi

About the author

Meital Matsafi

As head of community engagement at TestProject and a test automation enthusiast by ❤, I love building and maintaining relationships with community members, diverse community organizations and internal entities while learning and sharing my key findings.
Assisting with the development of robust engagement plans & activities I’m extremely eager to evolve and grow the community.

 

Join TestProject Community

Get full access to the world's first cloud-based, open source friendly testing community. Enjoy TestProject's end-to-end test automation Platform, Forum, Blog and Docs - All for FREE.

Join Us Now  

Comments

9 1 comment

Leave a Reply

popup image

Selenium for Teams

Sharing and distributing Selenium tests has never been so easy! With TestProject's FREE Selenium based platform, you can finally create awesome tests with the freedom to collaborate with your team effortlessly.
Sign Up Now right arrow
FacebookLinkedInTwitterEmail