If you’ve been thinking to yourself: “Hmm… which programming language should I start my testing journey with?” – Python is your answer. But it’s not just for beginners! In a recent survey I did on LinkedIn, we can see that even amongst experienced programmers, almost 35% voted that Python is their most favorite programming language (though not too far ahead from all-time favorite Java, that came in 2nd place with 32%).
Over the years, Python has indeed been gaining momentum among both testers and developers and it is the fastest-growing programming language out there (as we can see in the Stack Overflow Trends chart below), and we all know what that means… Popularity = relevancy!
By the end of this article, my goal is to help you see how powerful the Python programming language is, and which Python testing framework is the best fit for your project needs.
What’s So Great About Python for Test Automation?
Python is an open-source programming language that was first released in 1991 with an intent to provide an intuitive, simple, human-readable, elegant, and less-cluttered programming language to work with. And when it comes to testing, such qualities are a must, especially when we have manual testers in the team (that don’t necessarily have programming skills) who are now transitioning into automation and can enjoy Python’s fast learning curve to write scripts quickly.
There are quite a few reasons as to why Python’s popularity has been rising in the test automation field and why it’s thought of as one of the best choices for test automation. Amongst those reasons you will find: The Zen of Python (19 guiding principles for Python’s design philosophy), friendly for beginners but also powerful for experts, it’s both object-oriented and functional, it has a rich library of packages for testing, a worldwide strong community, and much more! I recommend reading more about it on the awesome AutomationPanda Blog.
The Right Python Testing Framework For You
With the increasing use of Python, the popularity of testing frameworks based on Python is increasing as well. At first, it can be a bit confusing knowing which one to choose out of all the wealth of tools out there as each one has its pros and cons. That being said, each project and organization has different requirements and limitations, thus we need to take them all into consideration when selecting a tool that will be the best fit for us. I recommend reading these great articles on how to choose the right tool for you:
- How To Select The Right Test Automation Tool
- How to Select the Best Tool – Research Process
- Criteria for Selecting the Right Functional Testing Tools
And of course – why do pros and cons lists exist, if not for helping us get additional guidance about our favorite tools, all in one place! 😉 So, let’s dive deep into the top 8 Python testing frameworks out there, and explore both their benefits as well as shortcomings so you can get that much closer to selecting the ideal Python testing framework for your needs.
Pros and Cons of Top 8 Python Testing Frameworks
Robot Framework (RF) is an open source test automation framework for acceptance testing, acceptance test-driven development (ATDD) and robotic process automation (RPA). Its core is implemented in Python, but can also run on Jython (Java implementation of Python) and IronPython (Python for .NET framework). To run it you will need to install Python version 2.7.14 or higher.
- Based on Keyword-driven testing (KDT) approach, thus allowing us to easily create test cases using human-readable keywords (no coding experience required).
- Supports all operating systems (Windows, Linux, or MacOS), and all applications (web, mobile and desktop apps).
- Provides clear and user-friendly HTML reporting data (including screenshots).
- Rich ecosystem with a lot of APIs making it a highly extensible framework and allowing to integrate with any other 3rd party tool.
- Great community support and online resources.
- Parallel testing is not supported out of the box, but it can be achieved via Selenium Grid or via Pabot (a parallel executor for RF).
- For good and for bad it force you to work according predefined methodology, Initial learning curve might be a bit longer then usual for rookies
- Creating generic keywords might take longer then just writing coded tests
- Not easy to customize the reports.
- Does not have an actual if/else construct, but there are several ways to get the same effect (as can be explored in their official documentation)
Bottom line: If you are looking to implement a keyword-driven framework approach that will allow manual testers and business analysts create automation tests, RF is the solution for you, providing a variety of extensions & libraries and easy to use. However, if you are on the lookout for developing complex scenarios, as you will need to do some customizations that are not built-in to the framework.
Pytest is an open-source testing framework that is possibly on of the most widely used Python testing frameworks out there. Pytest is supports unit testing, functional testing and API testing as well. To run it you will need Python version 3.5 or higher.
- Allows for compact and simple test suites.
- Is highly extensible by using plugins, such as: pytest-randomly, pytest-cov, pytest-django, pytest-bdd.
- You can also add pytest html plugin to your project to print HTML reports with one simple command-line option.
- Can run tests in parallel using a Pytest plugin pytest-xdist. You can read more about it here too.
- Has a very large community.
- Supports fixtures helping you to cover all the parameter combinations without rewriting test cases, and are a great way to manage context between steps.
- Compatibility is not key for Pytest, as although you can easily write test cases with Pytest, you won’t be able to use those within any other testing framework due to the usage of Pytest’s unique routines.
TestProject is 100% FREE complete automation framework with cloud reports. Using TestProject, you can easily develop test automation for mobile, web or generic purposes with the Python open source SDK. It supports Python version 3.6 or higher, and supports both Pytest and Unittest frameworks. TestProject includes all the dependencies required as part of the single cross-platform agent executable (You can watch this webinar recording to get started).
- Single Agent executable that includes all the 3rd party libraries needed to execute and develop test automation for mobile, web and generic tests.
- FREE Automatic cloud based reports in HTML/PDF format (including screenshots).
- Execution history accessible through RESTful API.
- Always up-to-date with the latest and stable Selenium/Appium driver versions.
- Single SDK for Web, Android, iOS and Generic tests.
- Built-in test runner and reporting capabilities.
- Cross platform support for Mac, Windows, Linux and Docker.
- Large community and support: forum, blog and built-in live chat.
- The agent can run tests one test at a time, thus for parallel testing you’d need to use Docker Agents.
- The reports are available only on TestProject cloud, to store it locally you will need download a PDF report or use the RESTful API.
- The Agent needs internet connection to run/execute tests.
Bottom line: If you are looking for a single framework that covers your entire automation efforts end to end, then TestProject is definitely the one for you and a perfect fit for teams with mixed skill sets, from beginners to experienced automation experts.
PyUnit (Unittest) is a unit testing framework for Python that was inspired by JUnit. It is the default Python testing framework that comes out of the box with the Python package, and thus the one most developers start their testing with.
- Since it is part of the standard Python library there are no additional modules required to install – it comes out of the box with the Python package.
- Offers simple and flexible test case execution.
- Quick generation of test reports, both XML reports and unittest-sml-reporting.
- The intent of the test code sometimes becomes unclear, since it supports abstraction.
- A huge amount of boilerplate code is required.
- Since based on Junit, camelCase naming method is in use, instead of Python’s snake_case naming convention.
Nose2 is a successor to Nose and is basically PyUnit (Unittest), but with plugins. Nose2 extends PyUnit beyond its functionality with various plugins that add support for test execution, test discovery, decorators, fixtures, parameterization, etc.
- Easiy to get started with as it extends the PyUnit (Unittest) framework that comes out of the box with Python library.
- Includes a large number of built-in plugins that can make your testing faster and easier.
- Enables parallel testing using the mp plugin.
- Collects tests automatically, as long as you follow some simple guidelines for organizing your library and test code.
- Lack of extensive documentation which can hold you back if you are just getting started.
- Not that actively maintained as opposed to other frameworks.
Bottom line: If you are already using the PyUnit, you might as well give Nose2 a shot and explore how it extends PyUnit.
Behave is one of the most popular Python BDD (behavior-driven development) testing frameworks. Although it is not officially part of the Cucumber project, it functions very similarly to Cucumber frameworks.
- Allows writing test cases in readable language, leading to easy collaboration across teams with similar features.
- A large amount of documentation and support are available to help in getting started.
- Fully supports Gherkin language – thus the creation of feature files does not require any technical knowledge.
- Has Django and Flask integrations.
- There’s no support for parallel execution.
- Only for black box testing.
Bottom line: If your team follows a BDD approach, you have previous BDD knowledge (such as Cucumber, SpecFlow, etc) and are looking for black box testing – you should definitely check out Behave. You should also explore other Python BDD frameworks such as: Pytest-bdd, Lettuce (to be described below), Radish and more in this Python BDD testing frameworks comparison article. If you need more than black box testing, then you should look elsewhere.
- Supports Gherkin language, thus enables even non-technical team members to easily create tests using natural language.
- Similar to Behave, it is used mostly for black-box testing but can be used for more testing types. For instance, Lettuce can test various server and database behaviors and interactions.
- Lacks some of the feature richness of other frameworks, so more suitable for small projects.
- Its support and documentation do not seem to be maintained.
- Requires dedicated communication between all stakeholders of the project: dev, QA and managers in order to make sure the implementation is successful.
Bottom line: If you have a small BDD project, Lettuce is a great option for easy, and natural language test creation among all team members.
- Used for unit testing, integration testing and system testing.
- For those who are familiar with Unittest, Testify is very easy to get started with.
- Has extensive plugins.
- Similar to Nose2, Testify enables test discovery.
- Simple syntax to fixture method.
- Lack of extensive documentation, thus beginners might need to invest in finding relevant resources.
- Parallel testing is not easy to achieve.
Bottom line: If you have previous experience with Unittest, it will be pretty easy for you to adjust your existing tests to work with Testify, so it’s definitely worth checking out.
We’ve come to an end of this python testing framework comparison list, and now it’s time to choose the one which fits your requirements the most.
Are you looking for more of a BDD approach? Is it unit testing you are looking to achieve or more of functional testing? Does your team have technical or coding experience, or do they mostly comprise of beginners? These questions and many others are the ones you need to take into consideration while trying to make a decision. Each of the above frameworks has its advantages as well as disadvantages, and there is no such thing as good or bad, but rather suitable. Suitable for your own needs and product requirements.
I hope this article was able to help you not only in understanding how great Python is for automation testing, but also help you in your journey to easy Python testing!
What is your favorite Python testing framework? Feel free to share in the comments and include any other frameworks I might have missed!
Happy Python-ing 😊🐍