Create Parallel Automation Tests

When the number of test scripts increases to a point in which it takes too much time to execute, then the usually quick feedback we receive from automated testing starts to decrease. In some cases, automation batch runs can even take days to complete, and then we need to start asking the “hard” questions – Are all the scripts we run useful, should we reduce the execution time through various improvements in the framework, and so on. If you’ve changed one of these steps and it still takes a long time to execute the batch, what should you do? Well, this is where parallelization comes in. Adding parallelization to your automation is beneficial, since it helps you to get a prompt feedback during your code changes submissions, it speeds up the feedback loop, and thus allows to fix bugs earlier in the cycle. Reducing the test feedback time also gives a great opportunity to improve the automation suite.

 

Adding parallelization – More than just adding environments

Parallel runs require more than simply adding environments and running tests in parallel. In fact, before we can even talk about running tests in parallel, there are certain characteristics in the automation framework design that need to be ensured. Rushing into parallel execution can create flakiness and undue failures which would be hard to rectify without changes in the script’s design.

In addition, some form of Continuous Integration needs to be added as well. Running all of the scripts at one given time in the Software Development Life Cycle is not effective. Instead, scripts should be executed at different stages, for instance: While the developer is checking the code, when a patch is deployed for testing, during regression around releases and so on. This way, there’s a decrease in the amount of scripts that are executed at each stage, and thus we can get quicker responses. This concept has been discussed briefly in this video .

 

Test data should allow parallel execution

If the same data is being used by multiple scripts which happen to change the data, these scripts wouldn’t run in parallel. For instance, if use case A has steps one, two and three. Correspondingly scripts UseCase001, UseCase002 and UseCase003’s steps one, two and three respectively use the same record, running one after the other, then running them in parallel would not be possible. This specific test data utilization method is not recommended in any case and it does not allow running tests in parallel.

As a rule of thumb, all tests should use separate test data which is exclusive to that test. Sharing data across test scripts can cause the need for maintenance efforts. From the get go, do not use any piece of data in more than one script. This also ensures there is no need for tests to be run in any specific order, since they are all independent which is also very important for parallel execution.

 

Smaller tests

Creating one long script corresponding to a complete use case is more prone to failure, thus instead, one should break it down into multiple tests. If your scripts are independent from one another, most of the job is probably done. But don’t get into the trap of breaking three small inter-dependent scripts to create one long independent script. Smaller “bite sized” scripts add more robustness and makes parallelization a lot easier.

 

Project structure

Although the project structure is a no brainer, it should be mentioned. Usually it’s a great idea to cluster scripts around the application under test functionality. This grouping of tests makes it easier to select scripts to run and makes it more user friendly. For instance, tests for a specific screen combined in one group, followed by screens under a specific module on a higher-level group, and so on.

Adding tags to scripts can also make things easier, since that way the execution can be done through tag selection. For instance, running scripts with the tags ‘Use case’, ‘Screen A’ and ‘API’. This way, the classification becomes much more easier and we can have lots of permutations to choose from for selecting our batch execution.

 

Scheduling tests

There are various ways to run parallel tests. There are many commercial tools that allow to run tests in parallel. Alternatively, you can use Selenium grid, or utilize virtualization to create multiple environments, or create on demand containerized execution environments in the cloud based on Docker with Kubernetes and have multiple batch runs that way. There isn’t a “best way” to do it, it just all really depends. One must see what best suits the team’s dynamics, the batch size, future scalability aspects, expected life of the project, skill level, costs associated, etc.

Whichever route you choose, using a CI tool like Jenkins, Team City or Circle CI can add a lot of benefit. You can implement the parallelization without them too, but I wouldn’t recommend that. These tools allow for triggering batches on events, scheduling and selecting scripts to run, a consolidated place to control all the executions and one place to get the results from as well.

 

As all things related to automation, parallelization is an evolutionary process! Starting it too early or too late in the project might necessitate rework. However, the pre-requisites mentioned in this article need to be enforced from day one of building the automation framework. This foundation will go a long way in evolving the automation efforts, including parallelization.

What do you think of parallel testing? Please share your thoughts in the comments below  😎