7 Open-Source Test Automation Frameworks

As we enter the last quarter of 2017, TestProject’s team decided to wrap up for you the best open-source test automation frameworks out there, and to help you choose the right one for you!
Here are the pros and cons of 7 different open-source test automation frameworks:

1. Robot Framework

Robot Framework (RF) is a test automation framework for acceptance testing and acceptance test-driven development (ATDD). This framework is written in Python, but can also run on Jython (Java) and IronPython (.NET), and therefore is a cross-platform (Windows, Linux, or MacOS).

Pros:

• It simplifies the test automation process by utilizing the keyword-driven testing (KDT) approach, that helps testers create readable tests that are easily made.
• Has easy to use test data syntax.
• Has a rich ecosystem around it consisting of various generic test libraries and tools that are developed as separate projects.
• Has many APIs that make it highly extensible.
• Although it’s not a built-in ability, RF enables to execute parallel tests via pabot library or via Selenium Grid.

Cons: 

It’s not easy to customize html reports.

Bottom line: This cross-platform framework is highly recommended if you are targeting for KDT automation with a wide range of libraries and extensions. If you want to add new keywords (via RF test library APIs), a basic knowledge in Java/Python/C programming language is required.

2. JUnit

JUnit is a framework for unit testing of Java applications, used to write and run repeatable tests.

Pros:

• Tests are written in pure Java which is known as the leading programming language worldwide.
• Supports test-driven development (TDD).
• Enables to create your own unit test case suite.
• Integrates very good with other tools (for example, Maven) and with IDEs (for example, IntelliJ).
• Has history – So it has a large user base that makes it easy to find documentation on it.

Cons:

• If a mocking ability is required, one needs to add Mockito (or some other mocking library).
• Tests are not readable by non-technical people, since for example the methods names in JUnit are constrained by Java conventions.

Bottom line: In case you are looking to write unit testing for your java application, it’s probably the best choice. However, for functional testing or non-java applications, you should consider other solutions.

3. Spock

Spock is a testing and specification framework for Java and Groovy applications. It is based on JUnit.

Pros:

• Creates readable tests and supports plain English sentences, making it easy to read.
• Provides the surrounding context, thus easily enables to understand how to fix a failure.
• Has built-in mocking and stubbing capabilities.
• Supports data-driven-tests (DDT).

Cons: 

Requires knowledge in basic Groovy programming language.

Bottom line: If your application is based on JVM and you are targeting for BDD test automation with DSL, this framework is just for you!

4. NUnit

NUnit is a unit testing framework for all .Net languages. Originally being inspired from Junit, it is written entirely in C#, and has been completely redesigned to take advantage of many .NET language features.

Pros:

• Quick initiation and test execution.
• Comes with assertions and annotations.
• Enables parallel testing.
• Supports test-driven development (TDD).

Cons:

• It is not a cross-platform as it is used only for .Net languages.
• It doesn’t integrate into Visual Studio echo system, thus using it means more maintenance.

Bottom line: A nice open-source framework for C# unit testing, with a long history and good reputation. However, in case you’re already using .NET languages, you may consider MSTest.

5. TestNG

TestNG is a test automation framework for Java that is inspired by JUnit and NUnit, but includes improved and new functionalities (NG – Next Generation). It is designed to cover all the test automation categories: unit testing, functional test, end-to-end, integration testing, etc.

Pros:

• It is easily integrated into maven cycle.
• Gives the developer the ability to write flexible and powerful tests.
• Supports Data Driven testing (DDT).
• Annotations are easy to understand.
• Test cases can be grouped easily.
• Allows to create parallel tests.

Cons:

• Only supports Java, so you need to have at least a basic knowledge of Java programming language.
• You have to invest time in framework setup & design.

Bottom line: If you use Java, looking for end-to-end test automation framework and willing to invest some time in framework setup – you should definitely consider using TestNG.

6. Jasmin

Jasmine is a JavaScript unit testing framework. It is also known as a Behavior Driven Development (BDD) testing framework for JavaScript. It’s suited for websites, Node.js projects, or anywhere JavaScript can run on. It is mainly paired with AngularJS.

Pros:

• Additionally to JavaScript, it can run in Python and Ruby, which can greatly assist you if you want to run your client-side tests alongside your server-side ones.
• Is supported by many CIs (Codeship, Travic, etc.).
• Has built-in syntax for assertions.

Cons:

• In most scenarios it requires a test runner (such as Karma).
• Has difficulties with asynchronous testing.

Bottom line: Jasmin might be a perfect fit for your needs if you’re searching for a unified (client-server) unit testing solution.

7. Mocha

Mocha is a JavaScript unit testing framework, that runs tests on NodeJs. It is mainly paired with ReactJS.

Pros:

• Has its own test runner built-in.
• Supports asynchronous testing.
• Allows flexibility since you can use any assertion library (Chai, expect.js, Must.js, etc.) that fit your needs (as a replacement to Node’s standard ‘assert’ function).

Cons:

• Relatively new to the field (developed in 2012), which means it’s still changing and its user base and support might be lacking in some aspects.
• Provides just the base test structure, thus requires additional setup and configuration (may be an advantage for some).

Bottom line: If you are looking for JavaScript stand alone unit testing framework, Mocha is your go-to framework!

 

Which open-source test automation framework do you use? What are their pros and cons? Please join the discussion in the comment section below  😎