In this article I’ll show you how to create comprehensible and professional HTML reports for your C# Selenium WebDriver tests, using TestProject 100% free test automation platform.
Why we need HTML reports
Test automation reporting is the most important activity that will make the automated testing effort more visible. The reports can be shared with the team, the manager, or even clients.
Good HTML reports show what scripts were executed, the steps executed, tests execution times, which tests and steps passed or failed, and screenshots of failures. It is detailed enough to help understand why the failures occurred – especially if it is a failure in the test script or a failure in the application under test.
But using an effective reporting mechanism has other benefits, apart from seeing passed/failed results. In time, it can help increase test coverage, test quality, and in the end, improve the quality of the product being tested.
Why Selenium WebDriver is not enough
Selenium WebDriver is a collection of APIs that allow the automation of browser interaction, such as page navigation, clicking buttons, filling out forms, and other common actions a regular user might perform in a browser.
While Selenium WebDriver is the most popular tool used for web test automation, it cannot be used as a standalone tool to perform tests.
For a proper test automation framework, you need a test runner (think JUnit, TestNG, NUnit, XUnit, MSTest) and a good reporting tool. Selenium WebDriver alone does not provide the option to make assertions that verify that the functionality works as expected, and also does not provide reporting capabilities. You can read more about reporting solutions in this comparison article: The 8 Leading Test Reporting Tools for Selenium.
This is where TestProject comes in – providing complete test reports, that include test results, screenshots for steps performed, trends (number of tests executed on each test run, number of tests passed/failed), and test duration.
Setting up the environment for TestProject testing
The first step is to create a new (free) TestProject account. Once you have set up your account, download the TestProject Agent for desktop, for the operating system you are using – it is available for Windows, Mac OS X, and Linux (and even Docker!). You can do this from the Agents tab in your account. Here is where you also need to register the agent.
Make sure to give it a meaningful name, because the agents can be shared between team members. “Jane’s Agent” is a much better name than “Test agent” or “123”. Also start the agent before running any of the tests, otherwise the tests will not start.
TestProject allows test creation from the interface as well, by recording the browser/mobile actions, but in this tutorial, we will focus on writing our own tests using TestProject’s SDK.
Create your first C# test with TestProject SDK
Now to the fun part.
If you are reading this article, I’m guessing you already have Visual Studio installed, if not, go ahead and install it because it’s the IDE we will use to write our tests.
Create a new project in Visual Studio. The selected framework should be .Net Core, and you can work with any test runner you are comfortable with. For this example, we will use NUnit as our test runner, so simply select NUnit Test Project (.Net Core) from the projects list, name the project, and save it.
Next, add the TestProject NuGet package: right-click on the project name, Manage NuGet packages…, browse for The TestProject SDK, and install the latest version.
Let’s move on to writing the test. From here, there are very few differences between a Selenium WebDriver test in NUnit and a TestProject test.
A test class using TestProject needs to implement one SDK.Tests interface: IWebTest, IAndroidTest, or IIOSTest. Our tests will be web tests that run on Chrome, so the interface we need to use is IWebTest.
Here is a simple example of a test that logs in https://example.testproject.io/web/ with the user “John Smith” and the password “12345”, and checks that the login was successful by verifying the presence of the Logout button:
using OpenQA.Selenium;
using TestProject.Common.Attributes;
using TestProject.SDK;
using TestProject.SDK.Tests;
using TestProject.SDK.Tests.Helpers;
namespace NUnitTestProject1
{
public class LogInTest : IWebTest
{
ExecutionResult IWebTest.Execute(WebTestHelper helper)
{
var driver = helper.Driver;
//open the https://example.testproject.io/web/ webpage
driver.Navigate().GoToUrl("https://example.testproject.io/web/");
//Fill in the form with the username and password
driver.FindElementById("name").SendKeys("John Smith");
driver.FindElementById("password").SendKeys(password);
driver.FindElementById("login").Click();
//Verify that the login was successful
if (driver.FindElement(By.Id("logout")).Displayed)
{
return ExecutionResult.Passed;
}
return ExecutionResult.Failed;
}
}
Let’s analyze the code:
- The test uses Selenium methods: Navigate() to open the webpage in browser, SendKeys() and Click() to interact with the form, Displayed to check if the element is present on the page, and the elements are located by their IDs.
- The WebTestHelper class provides access to the driver and other web action resources.
- The last part is returning the test result – whether the test passed or failed. If the login was successful, then the Logout button (with ID “logout”) is visible on the page, and the test is passed, if not, it means the test failed. The ExecutionResults are enums from the TestProject SDK, and the available values are: Passed, Failed, and Suspended. The result will be shown in the generated report once the test is executed.
To run tests, you need to use the Runner class. You can add it to the setup method, so the runner is started before running all the tests in the suite. As I mentioned before, this is an NUnit project, so we’ll use the NUnit.Framework notations for the test setup, test teardown, and for running the tests:
using NUnit.Framework;
using TestProject.Common.Enums;
using TestProject.SDK;
using System;
namespace NUnitTestProject1
{
public class NUnitTest1
{
Runner runner;
private string DevToken = Environment.GetEnvironmentVariable("TP_DEV_TOKEN");
[OneTimeSetUp]
public void SetUp()
{
runner = new RunnerBuilder(DevToken).AsWeb(AutomatedBrowserType.Chrome).Build();
}
[OneTimeTearDown]
public void TearDown()
{
runner.Dispose();
}
}
}
You can obtain your DevToken from the TestProject application, in the Integrations tab, and then add it to the code, or better than hardcoding it in the project, set it as an environment variable and then use the Environment.GetEnvironmentVariable() method to get it. Then it should be passed as a parameter to the RunnerBuilder. Now TestProject will recognize the account running the tests and save the reports in that account, where you can access them any time, see them in HTML format or download them as PDFs.
Next, add the login test to this class, this time using the NUnit Test annotation:
[Test]
public void Login()
{
runner.Run(new LogInTest());
}
After you build the solution, you should see this test in the test Explorer panel of Visual Studio.
Next, run the test – if everything is set up well, the test should run successfully. Don’t forget to have the TestProject agent running before executing the tests! The test execution is quite fast, but if you go to the Monitor tab, you can see that you have 1 job running, while the test is executing:
You can also download the demo code from GitHub.
The HTML Reports in TestProject
Now when you open the TestProject app in the browser and go to the Reports tab, you can see the results of the test that just ran.
The test was successful, so the HTML report should display this status. Moving through the available charts in the Trends and Reports page of the HTML report, you can see other useful information about the project: the number of tests run, the number of executions, the test results, the result trends, the agent used (remember before, when you had to name the agent? This is where that name can be seen), the platform (Web, Android, iOS). The chart can also be filtered to display only the most recent test results, the default value is for the last 30 days, but other options are available, as well as the option to customize the range.
In our particular case, we only have one test run, containing a single test, so this information is displayed in the Trends and Reports.
Next, if you click on the project name, you will be redirected to a more detailed report for the project. This report shows all available runs (the date and time the tests were run), and statistics for each selected run:
On the right-hand side of the page, there is a detailed description of each step of the test, with its result.
Let’s make the test fail by changing the ID of the login button into a value that will not be found on the page. Replace driver.FindElementById(“login”).Click(); with driver.FindElementById(“login-invalid”).Click();
Now when you rerun the test, it will show as failed and the statistics will be updated to include this test result:
As you can see, the report offers a lot of detail, and the failure message is a Selenium WebDriver exception – NoSuchElementException. Now, we already know why this test failed, because we changed the ID intentionally, but in a real-life scenario, this information is very useful to help us understand what is causing the test to fail, and whether we need to make changes to our tests or the test identified a bug in the application.
Now that you know that the reports show the test output, you can change the value back to the correct one.
If you want the report to provide more information, you can add screenshots. For this, you need to use the TestReporter class, and then add a reporter step that will record the screenshot. The test will now look like this:
using OpenQA.Selenium;
using TestProject.Common.Enums;
using TestProject.SDK;
using TestProject.SDK.Reporters;
using TestProject.SDK.Tests;
using TestProject.SDK.Tests.Helpers;
namespace NUnitTestProject1
{
public class LogInTest : IWebTest
{
ExecutionResult IWebTest.Execute(WebTestHelper helper)
{
var driver = helper.Driver;
TestReporter report = helper.Reporter;
//open the https://example.testproject.io/web/ webpage
driver.Navigate().GoToUrl("https://example.testproject.io/web/");
//Fill in the form with the username and password
driver.FindElementById("name").SendKeys("John Smith");
driver.FindElementById("password").SendKeys("12345");
driver.FindElementById("login").Click();
helper.Reporter.Step("Logged in the app", "The login is unsuccessful", "The login is successful", driver.FindElement(By.Id("logout")).Displayed, TakeScreenshotConditionType.Failure);
//Verify that the login was successful
if (driver.FindElement(By.Id("logout")).Displayed)
{
return ExecutionResult.Passed;
}
return ExecutionResult.Failed;
}
}
}
The new step will:
- display the description (“Logged in the app”),
- display a failure message (“The login is unsuccessful”) or a success message (“The login is successful”),
- verify the condition (driver.FindElement(By.Id(“logout”)).Displayed),
- and take a screenshot if the condition is not met.
If you run the test now, it will pass and display the success message, but no screenshot will be taken.
If you change the password to an invalid one, say “1234”, the report will display the screenshot for the failed step and display the failure message:
Final thoughts
Using the TestProject SDK you can quickly obtain HTML reports for your automated tests, complete with success versus failure trends, screenshots, available from any browser, or downloadable in PDF format.