Writing good test automation code is software development. As a result, the test automation code should be treated exactly the same as the production level code. So we need a Definition of Done for test automation so that we can call a test complete. In this article, I outline the DoD (the ultimate test automation checklist) that I’ve used at some organizations to drastically transform their test automation efforts.
Definition of Done for an Automated Functional Test
- The automated functional test can be successfully executed in all application environments.
- The automated test has greater than 95% accuracy.
- Your test uses the page object pattern exactly as prescribed here.
- Your test does not contain any reference to interactions with HTML (no button clicks, no element locators and so on…)
- You left the code cleaner than you found it.
The automated functional test can be successfully executed in all application environments
I hope that this doesn’t sound silly? However, I have worked on and worked with projects where the test automation is capable of only running in one environment, like staging. That’s a really poor use of test automation if that’s the case.
One of the benefits of good test automation is that it can provide extremely rapid feedback regarding the quality of the software being developed. We can then use that feedback to decided if we want to move the code from one environment to the next. If our automation cannot run in all environments, then it can’t be used everywhere. As a result, it’s less useful in helping us to release software faster.
The other benefit of good test automation is that you can run a single script in multiple environments. As the environments get more complex and integrated with other technologies, we can run our tests to make sure that we didn’t introduce any weirdness. Being able to run in a single environment drastically decreases the return of our automation efforts.
The automated test has greater than 95% accuracy
This means that our automated test is only allowed 5 false results out of 100. A false result is when the test fails for any other reason besides an actual bug or a requirements change. These are the failures that we normally refer to as “flakes”. The test can also pass incorrectly, but this one is much harder to catch. If we run our test once per day, then that means within 100 days we’re only allowed 5 of such results.
Whenever we notice that a test keeps failing too frequently just quarantine it until we are ready to drastically improve its stability. If we keep following this pattern, we will be left with a suite of functional tests that are extremely stable and provide real value to our employer.
TeamCity used to make this process easy because it would identify flaky tests for us. Afterward, we can analyze the historical trends to see the pass rate of the test and determine if it should be quarantined. Other than TeamCity, I don’t know of other tools to make our job easy here. It’s usually a manual effort otherwise. But well worth the time.
This is likely the most important item of the checklist. I’ve spoken about this in-depth before and believe that fixing this problem in our code is paramount to any test automation success.
Your test uses the page object pattern exactly as prescribed here
Please, let’s start following the page object pattern as it was originally prescribed. There are so many resources out there that add to the confusion with different page object patterns and even other models such as Screenplay Pattern. We just need to put on the blinders and use the simplest and easiest approach to our test automation.
I have gone through the struggles myself where I created different versions of the pattern. They were all a sub-optimal use of my time.
Imagine if we were trying to make a delicious bowl of spaghetti. Rather than focusing on the ingredients and cooking them well, we spent our time trying to reinvent the tools used for the cooking process. We decided that the spoon and the pot were not sufficient and that we would create our own to cook this bowl of spaghetti. And what if we didn’t do the best job in reinventing these tools? As a result, our spaghetti came out worse than ever and we wasted a ton of time on nonsensical tasks.
Hence, rather than focusing on reinventing ideas like the page object pattern or creating our own test runners to read spreadsheets or creating our own BDD syntax, let’s use our time to help write the fastest, most stable, most valuable test automation possible.
Your test does not contain any reference to interactions with HTML
We never ever want to reference anything related to the HTML in our test code. No ifs, and, or buts about it. The reason for this is that when the UI of our software changes, we want a single place to go and update our test automation code.
Here’s a code example of a test that exposes both locators and interactions with the HTML (slightly modified from real code):
The issue here is that when (yes, when, not if) the UI changes, we will need to go and update every single line of code that referenced the locators that changed. If an entire page is redesigned, this whole test is done. Even worse is that this is a single test. There could be hundreds or thousands of others that reference the same exact locators.
If all HTML interactions lived in a page object, and our tests used those page objects, then we would only need to go to one place to update all of that information. This idea basically enforces the Single Responsibility Principle (please read about it).
Rather, we want our tests to only convey user behavior, as such:
[Test]
public void ShouldBeAbleToLoginWithValidUser()
{
_loginPage.Open();
var productsPage = _loginPage.Login("standard_user", "secret_sauce");
productsPage.IsLoaded.Should().BeTrue("we successfully logged in and the home page should load.");
}
Please note how we don’t have any reference to anything related to the HTML. All of that logic lives here:
public class SauceDemoLoginPage : BasePage
{
public SauceDemoLoginPage(IWebDriver driver) : base(driver)
{
}
private readonly By _loginButtonLocator = By.ClassName("btn_action");
public bool IsLoaded => new Wait(_driver, _loginButtonLocator).IsVisible();
public IWebElement PasswordField => _driver.FindElement(By.Id("password"));
public IWebElement LoginButton => _driver.FindElement(_loginButtonLocator);
private readonly By _usernameLocator = By.Id("user-name");
public IWebElement UsernameField => _driver.FindElement(_usernameLocator);
public SauceDemoLoginPage Open()
{
_driver.Navigate().GoToUrl(BaseUrl);
return this;
}
internal Dictionary<string, object> GetPerformance()
{
var metrics = new Dictionary<string, object>
{
["type"] = "sauce:performance"
};
return (Dictionary<string, object>)((IJavaScriptExecutor)_driver).ExecuteScript("sauce:log", metrics);
}
public ProductsPage Login(string username, string password)
{
SauceJsExecutor.LogMessage(
$"Start login with user=>{username} and pass=>{password}");
var usernameField = Wait.UntilIsVisible(_usernameLocator);
usernameField.SendKeys(username);
PasswordField.SendKeys(password);
LoginButton.Click();
SauceJsExecutor.LogMessage($"{MethodBase.GetCurrentMethod().Name} success");
return new ProductsPage(_driver);
}
}
If anything related to this web page ever changes, we only need to go to a single place to fix it. Remember, implementation details will always change. We will always update the UI to new technologies. The business case and flow are unlikely to change.
If a user needs to login to your app today then this is likely the case for the business, regardless of whether you are using Angular or COBOL. That’s why we place any HTML related stuff into a single place, the page object. As opposed to a dozen tests.
You left the code cleaner than you found it
Code rot is what happens over time as we work on software. Through this process, we continue to leave small messes everywhere. Over time, these messes become so unbearable that they begin to impact our work and progress, that’s code rot. At some point, we need to do a large refactor because the code is so rotten. We have all been there.
Can you imagine if our automation didn’t need a large refactor? What if it actually improved and became easier to use over time. That would be pretty fantastic, right? Hence, every time we touch the code, we should also clean something else up. Rename a variable, rename a method, split up a method, that’s all it takes.
Here’s an example of a test before being cleaned up. Are you able to understand what this test is actually checking?
[Test, TestCaseSource(nameof(DataSource))]
public void RespondToAllItems(string accNum, string dataSubject)
{
#region Parameters
//if (accessionNumber.Contains(TestContext.DataRow["AccessionNumber"].ToString()))
//{
string subject = dataSubject;
string loginId = DataLookup.GetLoginIdByAccessionNumber(accNum);
string username = DataLookup.GetUsernameByLoginId(loginId);
string password = DataLookup.GetPasswordByStateCode(
DataLookup.GetStateCodeByLoginId(username));
string schoolName = DataLookup.GetSchoolNameByUserAndLogin(username, loginId);
string sessionNumber = DataLookup.GetSessionNumberBySchoolAndLogin(schoolName, loginId);
int lineNumber = DataLookup.GetBookletLineNumberByLoginId(loginId);
#endregion
#region Test Steps
Assessment assess = new Assessment(
UseAdminPageToGoToLocation(accNum, loginId).Driver, true);
try
{
assess.AnswerNonReadingWritingItem(subject);
Reporter.LogTestStepAsPass("Responded to accession number " + assess.GetAccessionNumber()
+ " for Item Type " + assess.itemTypeString);
}
catch (Exception ex)
{
ePScreenshot.SaveContentScreenshot(assess);
exceptionString = ex.ToString();
throw new Exception(exceptionString);
}
// if (assess.itemTypeString == "Comp" ||
// assess.itemTypeString == "CompR")
// {
// assess.PageWait(0);
// }
//}
#endregion
}
Here’s that test after a little bit of love ❤. Are you capable of understanding what is being tested now?
public void RespondToAllItems(string itemNumber, string dataSubject)
{
var assessmentTestData = new StudentAssessmentTestData(itemNumber, dataSubject);
var studentAssessmentPage = UseAdminPageToGoToLocation(assessmentTestData.ItemNumber, assessmentTestData.LoginId);
bool gotRaiseHandError = studentAssessmentPage.AssessmentItem.AnswerNonReadingWritingItem(assessmentTestData.Subject);
Assert.That(gotRaiseHandError, Is.False);
}
public class StudentAssessmentTestData
{
public string ItemNumber { get; set; }
public string Subject { get; set; }
public string LoginId => BookletDataLookup.GetLoginIdByAccessionNumber(ItemNumber);
public StudentAssessmentTestData(string dataItemNumber, string dataSubject)
{
ItemNumber = dataItemNumber;
Subject = dataSubject;
}
}
That’s the difference between code that is rotting and code that is flourishing 🌼
If we can avoid code rot it will save us a lot of headaches. We will be able to write automation quickly and efficiently. Our code will be a pleasure to read. Our code will not rot and might actually live for decades.
Conclusions
As the software that is being developed, the test automation code requires a definition of done. This way you can ensure that everyone on our team is following top quality standards to create the highest quality test automation. Use this checklist to keep everyone on your team on the same page.