As we enter the last quarter of 2017, TestProject’s team decided to wrap up for you the best open-source test automation frameworks out there, and to help you choose the right one for you!
Here are the pros and cons of 7 different open-source test automation frameworks:
Robot Framework (RF) is a test automation framework for acceptance testing and acceptance test-driven development (ATDD). This framework is written in Python, but can also run on Jython (Java) and IronPython (.NET), and therefore is a cross-platform (Windows, Linux, or MacOS).
- It simplifies the test automation process by utilizing the keyword-driven testing (KDT) approach, that helps testers create readable tests that are easily made.
- Has easy to use test data syntax.
- Has a rich ecosystem around it consisting of various generic test libraries and tools that are developed as separate projects.
- Has many APIs that make it highly extensible.
- Although it’s not a built-in ability, RF enables to execute parallel tests via pabot library or via Selenium Grid.
It’s not easy to customize html reports.
Bottom line: This cross-platform framework is highly recommended if you are targeting for KDT automation with a wide range of libraries and extensions. If you want to add new keywords (via RF test library APIs), a basic knowledge in Java/Python/C programming language is required.
JUnit is a framework for unit testing of Java applications, used to write and run repeatable tests.
- Tests are written in pure Java which is known as the leading programming language worldwide.
- Supports test-driven development (TDD).
- Enables to create your own unit test case suite.
- Integrates very good with other tools (for example, Maven) and with IDEs (for example, IntelliJ).
- Has history – So it has a large user base that makes it easy to find documentation on it.
- If a mocking ability is required, one needs to add Mockito (or some other mocking library).
- Tests are not readable by non-technical people, since for example the methods names in JUnit are constrained by Java conventions.
Bottom line: In case you are looking to write unit testing for your java application, it’s probably the best choice. However, for functional testing or non-java applications, you should consider other solutions.
Spock is a testing and specification framework for Java and Groovy applications. It is based on JUnit.
- Creates readable tests and supports plain English sentences, making it easy to read.
- Provides the surrounding context, thus easily enables to understand how to fix a failure.
- Has built-in mocking and stubbing capabilities.
- Supports data-driven-tests (DDT).
Requires knowledge in basic Groovy programming language.
Bottom line: If your application is based on JVM and you are targeting for BDD test automation with DSL, this framework is just for you!
NUnit is a unit testing framework for all .Net languages. Originally being inspired from Junit, it is written entirely in C#, and has been completely redesigned to take advantage of many .NET language features.
- Quick initiation and test execution.
- Comes with assertions and annotations.
- Enables parallel testing.
- Supports test-driven development (TDD).
- It is not a cross-platform as it is used only for .Net languages.
- It doesn’t integrate into Visual Studio echo system, thus using it means more maintenance.
Bottom line: A nice open-source framework for C# unit testing, with a long history and good reputation. However, in case you’re already using .NET languages, you may consider MSTest.
TestNG is a test automation framework for Java that is inspired by JUnit and NUnit, but includes improved and new functionalities (NG – Next Generation). It is designed to cover all the test automation categories: unit testing, functional test, end-to-end, integration testing, etc.
- It is easily integrated into maven cycle.
- Gives the developer the ability to write flexible and powerful tests.
- Supports Data Driven testing (DDT).
- Annotations are easy to understand.
- Test cases can be grouped easily.
- Allows to create parallel tests.
- Only supports Java, so you need to have at least a basic knowledge of Java programming language.
- You have to invest time in framework setup & design.
Bottom line: If you use Java, looking for end-to-end test automation framework and willing to invest some time in framework setup – you should definitely consider using TestNG.
- Is supported by many CIs (Codeship, Travic, etc.).
- Has built-in syntax for assertions.
- In most scenarios it requires a test runner (such as Karma).
- Has difficulties with asynchronous testing.
Bottom line: Jasmin might be a perfect fit for your needs if you’re searching for a unified (client-server) unit testing solution.
- Has its own test runner built-in.
- Supports asynchronous testing.
- Allows flexibility since you can use any assertion library (Chai, expect.js, Must.js, etc.) that fit your needs (as a replacement to Node’s standard ‘assert’ function).
- Relatively new to the field (developed in 2012), which means it’s still changing and its user base and support might be lacking in some aspects.
- Provides just the base test structure, thus requires additional setup and configuration (may be an advantage for some).
Which open-source test automation framework do you use? What are their pros and cons? Please join the discussion in the comment section below 😎