logo logo

Cypress vs. Selenium vs. TestProject – 7 Browsers Speed Benchmark

Speed Benchmark - Cypress vs. Selenium vs. TestProject

Popular test automation solutions for web applications aim for speed and multiple browser support. But which one is truly the fastest? And which has the widest support for various web browsers? Cypress vs. Selenium vs. TestProject – HERE WE GO! 💥

The TestProject team accepted the challenge and performed a benchmark review by using an identical test case scenario and focusing on the execution speed of the 7 leading browsers: Chrome, Headless Chrome, Firefox, Headless FireFox, Edge, IE11, Safari, all while using and comparing between the following frameworks:

  1. Selenium – The most popular open source web driver that provides the ability to automate browsers and test web applications running inside.
  2. Cypress – Node.js based testing framework for web client-side testing.
  3. TestProject – FREE testing platform for teams, utilizing Selenium for web (& mobile) application testing. TestProject provides countless tools for developers and test experts to create scriptless automated tests (using a smart recorder), or to create coded tests with Selenium APIs by using Java or .Net Core.

Cypress vs. Selenium vs. TestProject – Which Will Rank at #1?

Well, for all of you who can’t wait till the end of the blog – The “Testing Oscar” for the fastest execution and best performance for the tested scenario goes to: Headless Chrome, Selenium WebDriver and Java with only 1820ms per execution!!! 🥇⚡ The second place goes to TestProject! And the slowest performance was observed with Cypress execution.

The results are clear in the chart below, demonstrating the fastest execution combination per framework (the lowest number means the fastest execution):

Cypress vs. Selenium vs. TestProject - Executions Speed

 

Browsers per Framework Execution Time – Benchmark Review

Let’s dive a bit deeper to see the breakdown of the browsers per framework execution time:

Execution Time on Chrome per Framework 

As seen in the chart below, the recorded test executed with TestProject had the fastest execution time in the Chrome category!

Cypress vs. Selenium vs. TestProject - Executions Speed - Chrome

 

 

Execution Time on Headless Chrome per Framework

As seen in the chart below, the test executed with Selenium WebDriver and Java had the fastest execution time in the Headless Chrome category. Cypress was far behind and seems to be lacking headless support (as also seen in the Headless Firefox category below). We’d expect them to be around 2-3 seconds …

Cypress vs. Selenium vs. TestProject - Executions Speed - Headless Chrome

Execution Time on Firefox per Framework

As seen in the chart below, when it comes to the Firefox category – Cypress takes the lead as the fastest framework! TestProject is in second place with Firefox execution speed.

Cypress vs. Selenium vs. TestProject - Executions Speed - Firefox

Execution Time on Headless Firefox per Framework

As seen in the chart below, when it comes to headless Firefox execution, TestProject takes the #1 spot as the fastest framework!

Cypress vs. Selenium vs. TestProject - Executions Speed - Headless Firefox

 

Complete Execution Time Comparison of Framework/Browser:

Framework/Browser

 

Chrome
Chrome
Headless Chrome
Headless-Chrome
Firefox
Firefox
Headless Firefox
Headless-Firefox
Edge
microsoft-edge
IE 11
IE11
Safari
Safari
Cypress (JavaScript)
Cypress
2701 13,772 3404 13,544 2680 N/A N/A
Selenium WebDriver (Java)
Selenium
2697 1820 6148 5763 6825 8723 2407
TestProject (Recorded Test)
TestProject
2593 2608 4407 4044 6051 8626 3236

**Important note: Some experts will claim that these frameworks are made for different purposes and use cases and this is probably true. However, we believe that they overlap in their basic functionality and that they have a lot in common, hence some organizations adopt only one of these solutions for web test automation projects.

Oh and not to mention that they ALL aim for speed! SO… We wanted to find out once and for all which one of them is the fastest! 🏃‍♂️✨

Benchmark Assumptions 

  • For this benchmark review, we’ve used the same single page web application for each test (available publicly here: https://example.testproject.io/web/).
  • Since the benchmark was executed using a real web application and in order to eliminate possible network limitations as much as possible – We executed each sample test 20 times each.
  • The execution time listed in the benchmark review table is the average time received from 20 of the total executions.
  • To compare apples to apples, all executions were performed using real data and within a scope of supported browsers (no mocking or stabbing).
  • The executed test included 14 steps, and performed the following actions: Login process with the same user name and password credentials, filling several mandatory fields, validating missing field (address), filling the missing address field, saving the data and validating it has been saved successfully.
  • The test for Selenium was written in Java, the test for Cypress was written in JavaScript and the test for TestProject was recorded using TestProject’s in-browser recording tool.

Cypress vs. Selenium vs. TestProject – Conclusions

Out of these 3 frameworks we’ve compared – TestProject and Selenium are the ones that provide support for the most complete set of browsers.

While working with Selenium standalone, you need to manually download and connect the web drivers in order to support multiple browsers. Whereas with TestProject, you don’t need to work hard to do so, since it’s all taken care of for you automatically once you install the TestProject Agent. All we needed to do is record our test once, and then had the ability to execute it on any desired browser.

The top conclusions we found are:

  • Selenium is the fastest execution driver when it comes to Headless Chrome.
  • TestProject is the fastest with Chrome and Headless Firefox (which is a bit surprising since it utilizes Selenium behind the hood) 😎
  • Cypress is the fastest with Edge, but needs better support for Headless executions.

Based on this benchmark review and your own experience – Which is your favorite solution for automating your web apps? Share in the comments below! 🙌🧐

 

About the author

Meital Matsafi

As the Head of Community at TestProject and a test automation enthusiast by ❤, I love building and maintaining relationships with our community members, diverse community organizations, and internal entities while learning and sharing my key findings. Assisting with the development of robust engagement plans & activities I’m extremely eager to evolve and grow the community.

Comments

11 1 comment

Leave a Reply

FacebookLinkedInTwitterEmail