Cypress vs. Selenium vs. TestProject – 7 Browsers Speed Benchmark

Popular test automation solutions for web applications aim for speed and multiple browser support. But which one is truly the fastest? And which has the widest support for various web browsers? Cypress vs. Selenium vs. TestProject – HERE WE GO! 💥

The TestProject team accepted the challenge and performed a benchmark review by using an identical test case scenario and focusing on the execution speed of the 7 leading browsers: Chrome, Headless Chrome, Firefox, Headless FireFox, Edge, IE11, Safari, all while using and comparing between the following frameworks:

1. Selenium – The most popular open source web driver that provides the ability to automate browsers and test web applications running inside.
2. Cypress – Node.js based testing framework for web client-side testing.
3. TestProject – FREE testing platform for teams, utilizing Selenium for web (& mobile) application testing. TestProject provides countless tools for developers and test experts to create scriptless automated tests (using a smart recorder), or to create coded tests with Selenium APIs by using Java or .Net Core.

Cypress vs. Selenium vs. TestProject – Which Will Rank at #1?

Well, for all of you who can’t wait till the end of the blog – The “Testing Oscar” for the fastest execution and best performance for the tested scenario goes to: Headless Chrome, Selenium WebDriver and Java with only 1820ms per execution!!! 🥇⚡ The second place goes to TestProject! And the slowest performance was observed with Cypress execution.

The results are clear in the chart below, demonstrating the fastest execution combination per framework (the lowest number means the fastest execution):

 

Browsers per Framework Execution Time – Benchmark Review

Let’s dive a bit deeper to see the breakdown of the browsers per framework execution time:

Execution Time on Chrome per Framework 

As seen in the chart below, the recorded test executed with TestProject had the fastest execution time in the Chrome category!

 

 

Execution Time on Headless Chrome per Framework

As seen in the chart below, the test executed with Selenium WebDriver and Java had the fastest execution time in the Headless Chrome category. Cypress was far behind and seems to be lacking headless support (as also seen in the Headless Firefox category below). We’d expect them to be around 2-3 seconds …

Execution Time on Firefox per Framework

As seen in the chart below, when it comes to the Firefox category – Cypress takes the lead as the fastest framework! TestProject is in second place with Firefox execution speed.

Execution Time on Headless Firefox per Framework

As seen in the chart below, when it comes to headless Firefox execution, TestProject takes the #1 spot as the fastest framework!

 

Complete Execution Time Comparison of Framework/Browser:

Framework/Browser	Chrome	Headless Chrome	Firefox	Headless Firefox	Edge	IE 11	Safari
Cypress (JavaScript)	2701	13,772	3404	13,544	2680	N/A	N/A
Selenium WebDriver (Java)	2697	1820	6148	5763	6825	8723	2407
TestProject (Recorded Test)	2593	2608	4407	4044	6051	8626	3236

**Important note: Some experts will claim that these frameworks are made for different purposes and use cases and this is probably true. However, we believe that they overlap in their basic functionality and that they have a lot in common, hence some organizations adopt only one of these solutions for web test automation projects.

Oh and not to mention that they ALL aim for speed! SO… We wanted to find out once and for all which one of them is the fastest! 🏃‍♂️✨

Benchmark Assumptions 

• For this benchmark review, we’ve used the same single page web application for each test (available publicly here: https://example.testproject.io/web/).
• Since the benchmark was executed using a real web application and in order to eliminate possible network limitations as much as possible – We executed each sample test 20 times each.
• The execution time listed in the benchmark review table is the average time received from 20 of the total executions.
• To compare apples to apples, all executions were performed using real data and within a scope of supported browsers (no mocking or stabbing).
• The executed test included 14 steps, and performed the following actions: Login process with the same user name and password credentials, filling several mandatory fields, validating missing field (address), filling the missing address field, saving the data and validating it has been saved successfully.
• The test for Selenium was written in Java, the test for Cypress was written in JavaScript and the test for TestProject was recorded using TestProject’s in-browser recording tool.

Cypress vs. Selenium vs. TestProject – Conclusions

Out of these 3 frameworks we’ve compared – TestProject and Selenium are the ones that provide support for the most complete set of browsers.

While working with Selenium standalone, you need to manually download and connect the web drivers in order to support multiple browsers. Whereas with TestProject, you don’t need to work hard to do so, since it’s all taken care of for you automatically once you install the TestProject Agent. All we needed to do is record our test once, and then had the ability to execute it on any desired browser.

The top conclusions we found are:

• Selenium is the fastest execution driver when it comes to Headless Chrome.
• TestProject is the fastest with Chrome and Headless Firefox (which is a bit surprising since it utilizes Selenium behind the hood) 😎
• Cypress is the fastest with Edge, but needs better support for Headless executions.

Based on this benchmark review and your own experience – Which is your favorite solution for automating your web apps? Share in the comments below! 🙌🧐