- Selenium WebDriver
It would be hard to find a more popular web application testing framework than Selenium. Without the need to set up a server to perform tests (a significant upgrade from Selenium 1) the framework is easy to work with, as it opens a browser instance and controls it directly.
Selenium (2.0) has been operating since 2009, and in 2020 is still more than relevant and useful (And let’s not forget about Selenium 4 Alpha with a bunch of new features and capabilities!). One of Selenium’s strongest advantages is the fact of it being open source and working across all main browsers (Mozilla, Chrome, Headless, IE and more). This makes it the first choice for engineers starting their adventure both with test automation and web testing.
Open source from the start – Selenium has to date gathered a huge community that keeps growing and constantly delivering new solutions. Community development has its obvious disadvantages, but nothing a bit of patience (waiting for your query to be community solved) or hard work (finding third party plugins and creating custom libraries) won’t fix.
Perhaps the most inconvenient thing for Selenium is its scalability issue, but no worries there – if you need to run parallel tests, you’ve always got the Selenium Grid to work with. Here again, a bit of elbow grease will get you sorted out just fine.
What can you expect of TestProject’s open source automation framework? The support and the collaboration capabilities are nothing short of amazing here. No questions left unanswered (for long), ample documentation, tutorials, FAQs, and articles on all subjects make the platform easy to set up and work with.
TestProject offers a very well thought out UX and a single interface for testing both native mobile and web apps. The compatibility with Selenium and Appium makes the collaboration easy and appealing to everyone, speeds up the work, ensures high quality of tests, and most importantly – lets you develop native tests quickly and easily.
You can also choose from an ever growing set of powerful addons to incorporate in your tests, and create your own ones with the OpenSDK.
Compared to other frameworks, setting up mobile testing is highly reduced in complexity. Cross platform capabilities let you work not only across all major browsers, but also to test iOS on Windows (!).
Free cloud dashboards and analytics make sharing your results easier than ever before. Every test execution automatically creates an HTML report to go with your code, providing well documented and hassle-free E2E experience.
There’s little to say against adding TestProject to your toolset. Joining the community will make your work faster, easier and fun.
Puppeteer is a Google supported and sponsored project. It will come as no surprise this one is also Chrome-based. For now, you can only work with Google’s flagship and Chromium, but Firefox is well into implementing Puppeteer, so it shouldn’t be long before you can work with Mozilla’s browser as well.
The first advantage of Puppeteer is its Google connection – the search engine giant provides thorough support and documentation. The framework has been also gaining in popularity and growing a community smaller than other frameworks, but still considerable.
The installation of Puppeteer is quite seamless and the tool comes bundled with a fully compatible version of Chrome.
Google’s API for Chrome/Puppeteer interactions is quite intuitive, making the execution of tests a breeze, compared to many other frameworks. Although by default Puppeteer runs in headless mode, you can reset this if you want to monitor the execution live in non-headless.
All in all, Puppeteer is a very good and promising tool, with good reviews and most importantly – owned by Google, which ensures great support and future development.
📍 More on the differences between Puppeteer and Selenium can be found here: Selenium vs. Puppeteer – When to Choose What?
Next one up in cross-browser testing tools is Cypress.io. Many developers will tell you this end-to-end testing framework beats Selenium hands down, but I don’t see the Selenium community getting any smaller. Besides you’d expect that from a tool you need to fork out some money on, that is if you want a decent amount of test recordings and email support.
📍 Here’s an interesting benchmark article you might find interesting to this topic: Cypress vs. Selenium vs. TestProject – 7 Browsers Speed Benchmark
What makes Cypress so good then, you ask? The main thing is, it executes in the browser, on the device. No waiting for the network to catch up.
Generally, the main advantages of Cypress have to do with time savings, and in software, development time is most certainly money, so it is definitely a tool you will want to consider. With Cypress you save time pretty much every step of the way. For starters – it comes with Chrome browser bundled, so no tedious environment setup. Another thing is the documentation – very good and thorough, again much quicker to learn than from scattered community resources.
Back to testing itself, in Cypress you use assertions to define the desired state of the application and it automatically waits for that state before continuing. No setting up waits in the script anymore (like you would in Selenium). Cypress automatically waits for DOM elements to appear and only then executes.
Generally, Cypress is your choice when you’re pressed for time, and don’t mind paying a bit extra or being restricted to Chromium-based browsers.
Its feature rich palette enables eg. easier than in most tools asynchronous testing, with a clear and simple API. A little example of making things easier – Mocha identifies the fact of “done” function not being called, accompanied by no pending async tasks, as a synchronous test. A set of added syntactic features also makes the framework more open to running a wider variety of scenarios.
Mocha is open source, which may be an issue to those trying to find quick solutions to uncommon issues, but on the upside offers one of the most controllable test environments out there. Tweaking it up with a few right plugins for assertions (chai seems a good pick – not just for the hot drink name familiarity), and mocks and spies (here sinon seems the weapon of choice by popular vote) will make Mocha grow on you instantly.
Super easy to set up by simply installing into an empty directory, made even simpler for your react-create-app projects, where the Jest package is already bundled in, so you can start testing straight away.
Next comes the testing speed. Parallelization, caching babel transforms and prioritizing slowest tests, Jest optimizes the CPU usage to the max. It’s complete with matchers and spies, but what makes it really stand out is Jest’s extensive mocking library. Replacing dependencies with mock functions is perfect for speeding up your tests and retaining better control over parallelization. Aside from timer mocks, ES6 class mocks, you can even place your own whole mock modules (instead of just functions) in the mocks subdirectory and test interchangeably with the actual module.
Jest also supports TypeScript with its ts-jest package, and lastly – enables test coverage reports where you can generate data on how much of certain parts of the system were tested. Jest is well worth considering, especially for all your React, React Native app testing.
📍 Interested to learn more about React Native app testing? A couple months ago I did a comparison piece you might find interesting about the Top 5 Cross Platform Mobile Development Frameworks.
In the world of end-to-end testing of Angular apps, Protractor is pretty well known and appreciated, but aside from Angular it can also be used for regression tests of other-than-Angular web apps.
Protractor executes tests in the browser, thus mimicking actual user behavior and providing a more real-life test cases, while still automated. Protractor might be a bit of a pain to set up and configure as it’s a Node app, but that is also its biggest advantage, as it gives you access to huge variety of Node’s packages. The framework also offers automatic waiting, so you don’t need to add sleeps or waits to your tests. Protractor deals with it for you by executing steps as soon as a page’s pending task is finished, thus speeding up the tests.
The obvious positive of Protractor is E2E testing, letting you check the workings of your web apps in the most “lifelike” environment. The framework is often referred to as essential for Angular apps. This is not without merit as compared to other libraries and tools, Protractor offers exceptional speeds, connects with Angular.js framework, with built in support for a number of Angular’s elements, and allowing for parallel testing in several browsers.
📍 First React, now Angular – Is there one better than the other? You might find this “battle” of the frameworks interesting: Angular vs. React: Which One to Choose?
Jasmine is open source, and due to its versatility and good set of features, gathered a significant community around itself. You can use the framework either in a standalone mode where it adds all the required directories and files to your project folder, or with a test runner (like Karma – see below). In the first case, you will be able to run tests and view results in the browser, while in the second, you will need Istanbul to generate reports.
If you’re after testing Jquery frontend code, Jasmine’s node extension package will come in handy with a set of matchers for the Jq framework and HTML, CSS, and JSON API handler.
Angular’s official documentation suggests Jasmine for testing, and rightly so, as it’s easy to both use and integrate for Angular developers. Although Jest is catching up on popularity and community growth, Jasmine still holds the upper hand, being more seasoned, trusted and actually the main inspiration for Jest’s creators.
Now here’s one testing tool that doesn’t really belong in the “framework” category. To utilize Karma‘s potential, you will still need a proper testing framework – eg. the above described Jasmine. Karma is an HTTP server launcher for your HTML test runner file. The tool does have it’s value in increasing the reliability of your testing strategy, as it runs tests on actual devices and browsers. This comes useful when you want to make sure your app will behave consistently for all users regardless of the differences in the DOM implementations in different browsers. Testing across a variety of physical devices – phones, tablets, also comes in handy in ensuring consistency. Karma is thus used to ensure your app works the same way on a variety of browsers and devices.
Another end to end testing framework to consider for your browser-based web apps is the node.js powered Nightwatch. Although it receives mixed reviews, it’s worth mentioning for a few reasons. First of all – it offers clear and powerful syntax enabling quick and effortless writing of tests. You can also set up page objects quickly and consistently, with the framework discovering the declared objects in the directory. Nightwatch separates the HTML selectors from page objects with its own layer of abstraction, making your testing grounds more tidy and organized.
There’s actually not much more to say about the framework, although continuous integration with Jenkins, Hudson and TeamCity as well as working with some Cloud testing providers might sway the scale in its favor for some. Conversely, lack of TypeScript support might be one on the negative side when looking for a comprehensive testing tool.
One of the top advantages of the framework is its strive for simplicity. This goes both for writing commands in highly readable, concise and simple syntax, and integration with Selenium. Webdriver.io connects with Selenium via simple Requests calls to Selenium (2.0) REST API, instead of having its own custom integration.
Webdriver.io is easy to start, the configuration is simple and fast, and it works with many popular libraries (eg. Cucumber, Mocha and Jasmine). The community is strong and committed to supporting the framework, which makes it an interesting choice.
Now this (relative) newcomer to the test automation landscape was developed and is maintained by Microsoft. Playwright is built on top of Puppeteer, with the goal to address the issues of slow test execution, unreliable wait scenarios, and excessive boiler-plate code for browser setup.
The idea behind Playwright is to provide a tool explicitly for web automation, that will be fast and reliable enough for modern web.
As for cross-browser compatibility, Playwright supports Chromium, Firefox and Webkit, so you’ve got the main players covered. As for testing across browsers, you will only write the test once and run it across all three with practically no changes to configuration.
There are a few things that make Playwright stand out. Auto-wait for elements to be ready – this little feature will relieve you from writing some of the test code. Playwright will also emulate permissions, geolocation and even mobile devices. For this, as well as for parallel testing, Playwright introduces browser contexts. Meaning, you can use a single browser instance to create multiple, concurrent, isolated browser contexts.
All in all, Playwright is highly modular and will fit in perfectly with the rest of your stack. So well worth giving it a go.