4 Helpful Principles When Starting with Automated Regression Testing
Many times I get asked questions in the software testing field. Recently, I have been asked a question about automated regression testing: ‘How do I start creating an automated regression testing pack for a website which has been in production for more than five years?’
Once a website has been functioning and serving live customers for a number of years, then it is in a mature state. By mature, I mean there are (hopefully) no obvious serious bugs in the system and if any, they will be the subtle or edge case issues that are not easily spotted by everyone.
What we shouldn’t do, is to try to retrospectively write automated tests for all the stories that have already been developed and have become part of the system. However, what we want is a set of key scenarios that exercise the system end-to-end to ensure that future developments won’t jeopardize the existing functionality.
The steps below are some guidelines that can be used for an existing and already established website, in order to find the key scenarios and a method of expanding on these to create a functional regression pack:
First, you need to familiarize yourself with the website and its features. Start with exploring the site and learn its behavior. While doing so, you can also create a mind map of the structure of the website, the list of pages and the features in each page.
Mind maps are a great way of getting a high-level snapshot and overview of the whole website. We can always refer to the mind maps to get an understanding of how the pages are connected.
2. Gather Metrics
Gather site usage metrics from the marketing and/or analytics team. Most businesses embed “tracking tags” such as Google Analytics on their website to be able to track how users use the site. There is an abundance of information about the user behavior and common user journeys that can be retrieved from these tracking systems.
The reason why we need to gather this information is to be able to prioritize what test scenarios to automate first so that we get the most value in the shortest possible time.
3. Key Scenarios
Start with automating the core end-to-end scenarios through the web application. This will form the basis of our “smoke regression pack”. For example, for a typical e-commerce web application, the core end-to-end scenario is:
Homepage> Search results > Product details > Customer login / Register > Payment details > Order confirmation
It is important to note that, to start with, we only need to ensure that we can get through the pages, starting from the Homepage and reaching the order confirmation page. The purpose is to check that the purchase flow is not broken, rather than checking each page’s functionality in great detail.
Once we have the simplest and most common user flow covered, we can then look into more variations. Despite the numerous combinations of features and pages, one would notice that there are really only a handful of user journeys through the system that need to be considered.
Scrutinizing analytics data, you will probably find 80% of users would go through the same paths but with different data. Therefore, our smoke regression pack should be built based on these scenarios.
4. Increase Coverage
**Note: here I’m not talking about test coverage; the focus is on feature coverage.
Expand on the smoke regression pack to create a more extensive function regression pack by utilizing the mind maps and applying state transition testing technique to build the scenarios.
Entry Points – To begin with, we first need to find the entry points into the system. These entry points could be a user landing on the home page, a product details page, or a SEM (Search Engine Marketing) specific page.
Once we identify a particular landing page, we need to see what features there are on that page which the user can interact with. This is where mind maps become very useful. We have a high-level overview of the page and its features.
Here, the meaning of feature is either a single component like a sort option drop-down box or filling up a user details form or as simple as clicking a link.
Initial State – When we first land on an entry point in the application, there will be a state associated with that page. We record that as the initial state of the application. Whenever we interact with any of the features on that page, we are most likely going to alter its initial state.
Trigger – Some features, when interacted with, will either load the same page (e.g. sort options will keep the same page, but data will be sorted) or transition to another page (e.g. submitting valid user credentials). The thing that causes this transition, either to the same page or to another page, is called the trigger, such as the submit button.
Assertions – Whenever the state of the application is changed, by interacting with a feature, we need to make assertions to check the status of the new state. For example, when we submit a login form with valid user data, we need to assert that the user is now logged in.
We can continue with the same manner on the new transition, or go back to the initial state and interact with another feature until we cover all the important features of the mind maps.
Over time, the level of confidence in deploying new code increases as more scenarios are automated and run on a regular basis.
-I would love to read your software testing experiences in the comments, or questions about automated regression testing you might have. 🙂