Running Automated Tests in Parallel with TestNG

Running automated tests in parallel can be a time saver. When we have TestNG tests, we have multiple options for running our tests in such a way. We only need to perform a few simple configurations, as I will show in this article. There are 2 aspects I will be discussing: the first one refers to using data providers, while the second one implies that you are grouping and running tests using XML suite files.

Using data providers

Whenever a test method receives parameter values through a data provider method, we annotate the provider method with the ‘@DataProvider’ annotation. By default, we don’t specify any attribute to this annotation. With such a configuration, we are going to have the test run sequentially. This means the test method is invoked once, with one of the parameter values at a time. Once this current method + value combination finishes running, the next value is assigned to the test method and the test runs.

If in turn, we would like for the values to be passed to the test method in parallel, so that the same method but with different values runs in parallel, we can update the ‘@DataProvider’ annotation. For that, we only need to add the ‘parallel’ attribute, and assign it the ‘true’ value, as a String:

@DataProvider(parallel = true)

Now, each time we will run a test whose parameters are coming from this data provider method we just updated, the tests will run in parallel. By default, in TestNG, the maximum number of tests that will run in parallel that are using this data provider is 10. We cannot change this number directly in the @DataProvider annotation.

However, if we are using XMLs for creating test suites, there is something we can do there. We can set a suite attribute named ‘data-provider-thread-count’ to the value corresponding to how many tests we want to run in parallel when using the data provider. This attribute can be found at the ‘suite’ tag level. If let’s say, we would want to set this attribute to the value 4, the ‘suite’ tag would look something like this:

<suite name="Name of suite" data-provider-thread-count="4">

Suite level parallelism configuration

Whenever we have XML configuration suites files, whether or not any test in the suite is using a @DataProvider, we can apply parallelism to the tests in the suite. Looking at a simple XML configuration suite, it is made up of: one ‘suite’ tag; one or more ‘test’ tags inside the ‘suite’ tag; one or more ‘package’ or ‘class’ tags inside each ‘test’ tag. A simple example of a suite file using packages would be:

<suite name="Parallel Suite">
    <test name="First test tag">
        <packages>
            <package name="firstPackage"/>
        </packages>
    </test>
    <test name="Second test tag">
        <packages>
            <package name="secondPackage"/>
        </packages>
    </test>
</suite>

Now, in order to assign parallelism to the tests, we can either do that on a ‘suite’ tag level or a ‘test’ tag level:

• on a suite tag level, we can have parallelism so that:
  • we run each ‘test’ tag in parallel – that means that everything inside the test tag will run sequentially, but the test tags themselves will run in parallel. For the example above, this means that we are running, in parallel, packages “firstPackage” and “secondPackage”, but from each package, only one method of one class is running at a time. To achieve this, at the ‘suite’ tag level, we need to add the attribute ‘parallel’ with value ‘tests’:

    <suite name="Parallel Suite" parallel="tests">

  • we run classes in parallel – this means the ‘test’ tags will run sequentially, but whenever the current ‘test’ tag runs, the classes inside the ‘test’ tag will run in parallel. To achieve this, at the ‘suite’ tag level, we need to add the attribute ‘parallel’ with value ‘classes’:

    <suite name="Parallel Suite" parallel="classes">

  • we run methods in parallel – this means the ‘test’ tags will run sequentially, but whenever the current ‘test’ tag runs, the methods inside the ‘test’ tag will run in parallel. To achieve this, at the ‘suite’ tag level, we need to add the attribute ‘parallel’ with value ‘methods’:

    <suite name="Parallel Suite" parallel="methods">

• on a test tag level: similarly with what we saw for the suite level parallelism, we can add, for each test tag, parallelism for either classes or methods. This is achieved using the same attribute and values as before. Either:

  <test name="First test tag" parallel="classes">

  or

  <test name="First test tag" parallel="methods">

Of course, you could think about combining these two: ‘suite’ tag level and ‘test’ tag level parallelism. And maybe some of the tests inside one of your packages or classes also have parallel data providers. However, too much parallelism can lead to unexpected results and unpredictability. Especially if you have dependencies between tests, possibly data used by each test, where each test updates this data. So, use this mechanism carefully.

One additional thing we can specify in the tag where we assign the parallelism is the number of threads we want to use for the parallelism. By default, this number is set to 10. If we want to change it, we just need to add the ‘thread-count’ attribute with its corresponding value. When setting a different number of threads, just keep in mind that having too many threads running in parallel can considerably degrade the performance of the machine where these tests are running, thus causing them to run more slowly.

Conclusion

As we have seen in this article, our tests can easily be configured to run in parallel with TestNG. We can try different settings to see what is the best way to configure them so that they run efficiently and predictably. And, even though parallelism is a nice and powerful mechanism, we need to use it wisely 😉