Test Automation Strategy for Your Agile Project

In this article, I’ll list down the key points to consider when starting to plan your test automation strategy for your agile project in order to get the most out of your test automation endeavor!

Executive Summary

Automated testing is a core activity of any agile development methodology (read: advice of an experienced QA Manager about agile testing). As we move towards continuous deployment, test automation becomes even more important due to the quick feedback response that it provides to the development team on the well-being of the application.

In order to get this quick feedback, automated tests need to be executed continuously, should be fast and test results should be consistent and reliable. In order to achieve these goals, the majority of verifications should be completed as a part of the development of new features. In other words, development and testing should be a coherent activity and quality should be “baked in” right from the start by ensuring that what is being developed works and that it didn’t break any existing functionality.

This requires “inverting the test automation pyramid” by pushing down GUI tests that take a long time to execute, to lower levels e.g. API layer that can run straight after the unit tests as part of the build to provide the initial level of confidence.

Test Automation Strategy Overview

Prevention rather than detection – while every effort should be spent on preventing the introduction of defects in the application in the first place, the techniques and methods for that are outside of the scope of this post. Here, the methodologies are defined to allow quick detection of bugs when they are introduced in the system and feedback to development.

Quality should be favored over quantity, in most cases it is better to release one feature that is rock solid rather than multiple features that are flaky. When releasing, as a minimum requirement is to not have any newly developed features with regression defects.

As already mentioned, quick feedback on the well-being of the application is of huge importance to support continuous delivery. Therefore, a process and a mechanism by which we can obtain feedback quickly needs to be created.

One way of getting quick feedback is by increasing the number of unit tests, integration tests and API tests. These low level tests will provide a safety net to ensure the code is working as intended and helps prevent defects escaping in other layers of testing. Unit tests form the foundations for test automation at higher levels.

The second element of improvement is running regression tests more frequently and aligned with the process of continuous integration, see later on. Automation testing should not be seen as an isolated task, but rather as a coherent activity embedded in the SDLC.

Automated Regression Testing

Automated regression testing is the core of the test automation strategy. Definitions of different regression packs:

Smoke Regression Pack, which is a sanity check that the application can be loaded and accessed. Also, just a few key scenarios should also be run to make sure the application is still functional.

The aim of the smoke test pack is to catch the most obvious issues, such as if the application isn’t loading, or a common user flow cannot be executed; for this reason, the smoke tests should last no longer than 5 minutes to give quick feedback in case something major is not working. The smoke test pack runs on every deploy and can be a mixture of API and/or GUI tests.

Functional Regression Packs, which are meant to check the functionality of the application in more detail than the smoke test.

Multiple regression packs shall exist for different purposes. If there are multiple teams working on different sections of the application, then ideally there should be different regression packs that can be focused on the area the team is working on.

These packs should be able to run on any environment as and when required, provided the behavior of the features remains consistent throughout the environments. They are executed multiple times a day and should last no longer than 15 to 30 minutes.

As these functional tests are more detailed, it takes longer to run them. Therefore, it is important to have the majority of functional tests at an API layer where tests can be executed faster so we could be within the 15 to 30 minutes time limit.

End 2 End Regression Pack, which tests the whole application as a whole. The aim of these tests are to ensure that various parts of the application which connect to various databases and third party applications, work properly.

The End-to-End tests are not meant to test all of the functionalities as those are already tested in the functional regression packs, however those tests are “light-weight” which are just checking the transitions from one state to another, and a handful of the most important scenarios or user journeys.

These tests are mainly executed through the GUI, as they are checking how users would use the system. The time taken to execute these can vary from one application to another but they are usually run once a day or night.

Test Automation Strategy for Multiple Agile Teams

 Automated Unit Testing

• Test automation starts at the unit level. Unit testing should be written by developers for any new feature that is developed. These unit tests form the foundation of a larger automation practice that spans all the way up to the System GUI tests. It is the responsibility of the developers to ensure that for every new feature that is developed, a set of coherent and solid unit tests are written to prove that the code meets the requirements and works as intended.
• Unit testing provides the most ROI to the team as they are very quick to run, easy to maintain and modify (as there are no dependencies) and when there are errors in code, it is quickly fed back to the developer.
• Unit tests are run on the developer’s machine as well as the CI environment.

Automated Integration / API or Service Testing

While Unit Testing is based around testing the functions within a class, Integration Testing forms the next level up from unit tests to testing the classes that collectively makeup the component to deliver a piece of functionality. These tests are executed only when the unit tests have run and passed.

Service Tests are naturally run at an API layer without the intervention of the GUI web interface; hence tests would be able to verify functionality in a pure form and because the tests talk directly to the components, they are fast to execute and will be part of the build.

Where necessary, mocks such as wiremock will be used to factor out the dependence of other 3^rdparty systems and when the downstream systems are not available to provide the data required for automation testing.

Integration Testing and/or Service Testing can be run on the developer’s machine as well and be part of the build, but if they start to take a long time, then it is best to run on the CI environment.

Tools such as SoapUI can be used for Service Testing.

Application Testing

A typical e-commerce application can be split into different applications or “apps” that provide different functionalities. The concept of Application Testing is that a group of tests that test the functionality of an app are organized together and run against the desired app. This pack will be useful in cases when a team wishes to release an individual app and would like to know whether it is functioning correctly.

The purpose of application testing is to ensure that features of the app are functionally correct. Due to the test being organised in a manner to indicate the health of a particular app, these tests are normally referred to as Vertical Tests, since they execute “down” a particular app. The tests are very thorough and coverage is large.

Selenium WebDriver could be used to run these automated tests against the browser. This tool is the most popular software testing driver for web automation to implement in your test automation strategy, which provides a rich SDK that allows complex verifications.

End 2 End Scenario Testing

The GUI automated tests which are run against the system, serve as typical user flows, journeys or end-to-end scenarios. Due to issues with this type of tests (discussed below) these will be kept at a minimum. The end-to-end scenarios are included in the nightly regression pack.

Inverting the Test Automation Pyramid

As part of the test automation strategy, we need to minimize the number of automated tests that are run at GUI layer. Whilst running automated tests through the GUI provides good and meaningful tests in terms of simulating a user’s interaction with the application, it is prone to many issues:

Brittle tests – since the tests rely on html locators to identify web elements to interact with, as soon as the ID change, the tests fail, therefore they bear a lot of maintainability costs.

Limited testing – the GUI could confine the tester’s ability to fully verify a feature as the GUI may not contain all the details from the web response to allow verification.

Slow page load time – because tests are executed through the GUI, the page load times can substantially increase the overall testing time and as such the feedback to the developers is relatively slow.

Least ROI – due to the issues mentioned above, the GUI automated tests provide the least ROI.

The automated browser tests should be kept at a minimum and will be used to simulate a user’s behavior incorporating common user flows and end-to-end scenarios where the system is exercised as a whole.

-There you have your test automation strategy for your own agile projects. Once you follow it, you’ll be able boost the efficiency and effectiveness of your projects. Which is the most crucial component in your test automation strategy?

Reference