We’ve often heard the term, “Regression Testing” and we may know that it’s done to verify that a code change in the software does not impact the existing functionality of the product.
But when should we do such testing, what are its benefits, and how can we perform regression testing well? In this article, we’ll dive into all the What, When & How’s of regression testing.
Table of Contents – All About Regression Testing
- What is regression testing?
- When should we do regression testing?
- How can we perform regression testing efficiently?
- Types of regression testing
- Regression testing challenges
- Regression testing benefits
- Summary – TL;DR
When you are making a salad, you wash it and add it to a bowl. You know it will taste good, but by adding some seasoning/sauces the taste changes. So, the question is does your pallet approve it or have you messed up a salad entirely? 🥗
Regression testing is very similar to the analogy above. It is about discovering and understanding the risks of the product under development. The responsibility of regression testing is to make sure the product is stable when code changes/updates are added to the source code. The development team should bear in mind that any alterations could cause adverse effects on the whole system and damage the product’s core functionalities. Therefore, with this type of testing, a tester is minimizing risks of dependencies or defects due to code changes.
As our teams grow, so does the product, and complexities rise, which leads to a greater effort and coverage for regression testing. As a team, we need to make sure the regression tests are not massive as this can impact delivery times. You need the right balance. Having a good plan and carrying out regression tests to discover if anything has changed in known areas of an application is a good start. The tests can also give you a better view of whether the code changes impacted the product’s quality. In terms of a timeline, regression testing is a final step to ensure the entire application performs as intended.
- A new requirement is added to an existing feature/implementation
- A new functionality is added
- The code is fixed to solve defects
- The source code is optimized to improve performance
- Patch fixes are added
- The configuration of an application changed
Regression testing examples:
- Bug fix regression testing: Retesting bugs to see if they are fixed.
- General functional regression: We retest the product broadly, including areas that worked before, to see whether more recent changes have destabilized the working code.
- Configuration testing: The program is run with a new device or on a new version of the operating system or in conjunction with a new application. The external components that the software under test must interact with.
- Localization testing: The program is modified to present its user interface in a different language and/or following a different set of cultural rules. Localization testing may involve several old tests along with several new tests.
- Smoke testing: A relatively small suite of tests is used to qualify a new build. It helps verify new implementations are built, deployed, and function as intended. You can also include some sanity testing just to make sure new features that are ready to release are working fine and/or check bugs verified are fixed.
I came across a super interesting regression testing heuristic created by Karen Johnson, which I found quite useful. It is known as RCRCRC, a mnemonic that stands for:
- Recent: new features, new areas of code are more vulnerable.
- Core: essential functions must continue to work.
- Risk: some areas of an application pose more risk.
- Configuration sensitive: code that’s dependent on environment settings can be vulnerable.
- Repaired: bug fixes can introduce new issues.
- Chronic: some areas in an application may be perpetually sensitive to breaking.
As testers, we are quite busy testing new features as they come… But when we are doing regression testing, it is vital to make sure we have a test suite. It is not an added value to exhaustively re-run all the tests. Have a planned target of what tests to run when performing regression. Some other things that can come in handy are:
Pairing: Pairing is something I fully support and look forward to. Pairing with a developer is useful as you will learn what changes are going into the product. You can then organize your regression plan accordingly. Read more about pairing from Lisa Crispin’s blog.
Understand your product: Do you fully understand your data and/or your application under development? It is very important you understand what is being developed, as this will allow you to create accurate regression tests and verify any high/low-risk areas in the application too. I suggest pairing as a good option to be clear on a product functionality.
Are you reviewing code? Code review is something usually developers do amongst them, but nothing stops testers from being involved. This also helps understand the new code that is going in and how it will impact your testing. You may learn a nugget of information by just questioning and listening which could help your testing strategy.
Release process: I like the idea of developers releasing code and testers deploying that to a test environment often and testing the new code. It is always good to release small and often, as loads of changes go through for a release that means more risks to test for.
If you are following a waterfall or agile methodology, then regression testing is slightly different. In a Waterfall methodology testers perform regression checks on new features plus older ones, basically run a full test suit. Whereas in an Agile team you release in small chunks and more often as the aim is to provide the client with the product sooner and learn from the client feedback. To understand these methodologies you can also read my previous blog.
Regression testing comes in different types and forms. How do you pick the one that works for you? Think of your software development life cycle and specific updates that you’re planning to introduce. Here is a list of the common types of regression testing techniques:
Corrective regression testing: Corrective regression testing is appreciated for its simplicity as it requires little effort. It reuses the existing test cases, given that no major changes in the product specifications have been made (no need to change a testing scenario).
Retest-all regression testing: As the name suggests, this is a complex testing approach that needs all the system’s components tested again from scratch. This test checks retrospectively every small change that’s been made since the very start.
Sometimes, this approach is used when suspicions arise that something was missed (or wasn’t done right) during the previous testing stages. Under such circumstances, it can be more effective to retest everything from scratch than attempting to identify what went wrong previously.
It’s not a very common approach, giving the budget concerns; but it gives you more quality reassurance about the system’s current state.
Selective regression testing: This type of testing can be viewed as a middle ground between corrective regression testing and retest-all regression testing: it’s not as basic as the first one, but not as comprehensive as the latter one. Selective regression testing picks critical test cases from a set to inspect the affected parts of the code.
Progressive regression testing: When the previously developed test cases are no longer useful, the newer ones get created. This is often the case when product specifications are modified, so new test cases must be created to address the changing requirements. Progressive regression testing involves using new testing scenarios when the product vision is affected.
Complete regression testing: Many companies using the Agile methodology that encourages doing fewer but frequent updates find that in real life, that’s not always possible: companies need to react quickly to the ever-changing client demands. Thus, large-scale system updates are possible (but not likely). But when they do happen, complete regression testing is performed: it’s a must when the root code has been changed.
Partial regression testing: When separate modules in the development cycle are merged with a shared code mainline, partial testing is performed, using primarily the existing test cases.
Unit regression testing: A straightforward approach that is focused on testing code as a single unit with all the dependencies, integrations, and interactions left outside the scope.
Following are the major testing problems for doing regression testing:
- With successive regression runs, the test suite becomes fairly large. Due to time and budget constraints, the entire regression testing suite may not be able to be executed.
- Minimizing the test suite while achieving maximum Test coverage remains a challenge.
- Determination of frequency of regression tests, i.e., after every modification or every build update or after a bunch of bug fixes, is a challenge.
How does regression testing benefit us all as a team? Remember regression testing is not just the testers’ responsibility, it’s a team responsibility.
- Regression testing executes the selected scenarios and ensures functionalities work smoothly and the application is not broken.
- Regression testing helps capture bugs prior to releasing or helps retest bugs that can impede releasing a new feature. Also something super important, if you have found a bug during regression testing, make sure this bug is for once and for all resolved. If this keeps appearing in regression testing, it means your application has something vital missing.
- Top-notch quality: the aim is to make sure an application is stable as well as getting enhanced continuously.
- Automation of regression test cases allows companies to keep the costs low by minimizing the team and involvement of human interaction. In the meantime, the team can focus on more strategic and important tasks. Or do some exploratory testing! ✨
- It allows a timely intervention. If a troubling area is present in the application, then solutions can be worked on as a team just in time, before it is released to production! It also makes the software reusable and maintainable.
It is vital to have a stable and maintainable product, therefore regression testing becomes a ritual in the team when releasing the product. But to be able to successfully do this there are some things to take into consideration such as good planning, team awareness, visibility, and testing approach.
Therefore pair up with developers and team members to understand the product and its user journeys as this will all help plan a better regression test suite and improve the product quality. As the above image explains, you may have fixed a bug and retested it, but in turn it may have also generated another bug, so it’s crucial to have a detective hat on! 🕵️♀️ Also, avoid any “Swiss cheese” situation and make sure high-risk areas are tested thoroughly during regression.
Remember a team that works together by communicating and collaborating leads to a much smoother testing process. Get involved as early as possible in the SDLC. Finally, regression testing is all but positive for a product as it acts somehow like a product safety net! 💪