Fastest Mobile Automation Testing Tool – Appium vs. XCUITest vs. TestProject vs. UiAutomator vs. Espresso

There are several highly popular frameworks for mobile test automation, so I went on a mission to find the fastest mobile automation testing frameworks and best the locator strategies out there. Is there even a noticeable difference? 🤔

I created a short benchmark overview running similar tests across iOS and Android apps, while using different frameworks and locator strategies. For this overview I’ve created two simple apps:

• One is an Android app created with Android Studio on windows.
• One is an iOS app created with Xcode on a macOS.

For those of you who can’t wait until the end for the results – Here’s a recap of this speed benchmark:

• Appium and TestProject are the only 2 frameworks that support both iOS and Android.
• TestProject was by far the fastest one on Android (92-119 milliseconds), and is hassle-free compared to the others since you just need to install a lightweight Agent and go ahead running your tests, that simple.
• Appium takes time to set up, but is fairly easy to use when it comes to coded tests. However, it was by far the slowest on Android.
• Espresso/UiAutomator/XCUITest were all fairly close in terms of speed, and all support a similar set of features. Easy to use in their respective environments, however all use very basic recorders, which means they can only be utilized for coded tests apart from basic actions.

Table of Contents

1. Benchmark Recap of all 5 Mobile Automation Testing Tools
2. Mobile Automation Testing Tools – Summarizing Diagram
3. Review of all 5 Mobile Automation Testing Tools
   • Appium Pros and Cons
   • XCUITest Pros and Cons
   • UiAutomator Pros and Cons
   • Espresso Pros and Cons
   • TestProject Pros and Cons
4. Speed Tests of all 5 Mobile Automation Testing Tools
   • Speed tests of UiAutomator vs. Espresso vs. Appium vs. TestProject
   • Speed tests of Speed Tests XCUITest vs. Appium vs. TestProject

---

Mobile Automation Testing Tools – Summarizing Diagram

Review of all 5 Mobile Automation Testing Tools

Before diving into our speed benchmark, let’s review the 5 chosen mobile automation testing frameworks:

1. Appium

Appium is an open source tool for automating native, mobile web, and hybrid applications on iOS, Android and Windows platforms.

Appium has client libraries in Java, Ruby, Python, PHP, JavaScript and C#, which support Appium’s extensions to the Selenium WebDriver protocol. This makes Appium highly versatile, albeit with a few caveats:

1. Installation can be a bit of a hassle. You need to either install all the separate packages and libraries on your own, or install node.js combined with something called NVM to help you manage the installation process.
2. iOS tests require Xcode, which is Apple’s proprietary IDE, so you’d need to either use a macOS, or Windows with Appium Studio (community edition), or a macOS virtual machine (perhaps not entirely legal). If you are testing on both Android and iOS, this can be quite tedious unless you own a mac.

Appium uses the vendor-provided automation frameworks, which means its using XCUITest for iOS devices, and UiAutomator for Android devices.

Pros and Cons of Appium

Appium Pros ✅	Appium Cons 👎
Free	Installation needs multiple steps, can be complex
Supports both iOS and Android	Can run both iOS and Android only on macOS; Windows only supports Android
Open source	Requires some technical skills
Inspector session is very simple and intuitive	Requires programming skills
If you like coded tests, it is convenient	Not easy for testing newbies

 

2. XCUITest

XCUITest is Apple’s proprietary framework. It requires Xcode to run, so it means you can only use a mac to run it, or use the “bypasses” mentioned in the Appium review above, but you cannot run it natively on Windows! It supports Swift and Objective-C.

XCUITest Pros and Cons

XCUITest Pros ✅	XCUITest Cons 👎
Convenient if you are familiar with iOS app development	Background in iOS app development required
Xcode has a very intuitive recorder	Recorder can only support very basic actions
Uses the same IDE you created your app on	Proprietary – You will need to learn the syntax if unfamiliar
Directly supported/updated by Apple	Can be used mainly for unit testing
Very good documentation and tutorials by Apple	This framework was built for developers rather than QA engineers
	Requires application source code/IPA

 

3. UiAutomator

A Google unit testing framework. It supports Java and Kotlin, and you can run it on any Java supporting IDE.

UiAutomator Pros and Cons

UiAutomator Pros ✅	UiAutomator Cons 👎
Convenient if you are familiar with Android app development	Background in Android app development required
Has a very intuitive recorder	Recorder can only support basic actions
Uses the same IDE you created your app on	Proprietary – You will need to learn the syntax if unfamiliar
Directly supported/updated by Google	This framework was built for developers rather than QA engineers
Very good documentation and tutorials by Google	Requires application source code/APK

 

4. Espresso

Espresso Pros ✅	Espresso Cons 👎
Frequent updates and provides enhanced support for newer Android versions	Need to build the APK before running tests
Tests live in the app and supposed to run much faster	Application source code is required, meaning this framework is mainly for developers
Provides Automatic Synchronization to make tests more stable	Waits for the main UI thread to be idle before test actions are performed
It launches fast	Doesn’t support webviews.

 

5. TestProject

TestProject is a next generation test automation platform, built on top of Appium and Selenium and aims to simplify the test automation experience. The biggest upside of TestProject is that it is one quick installation (that simplifies all of the Appium and selenium component installations and configuration), and you are good to go for testing on Android, iOS (even on windows!) and Web on any operating system (Windows, macOS and Linux). Here are some additional features that make TestProject perform better than others:

• Simple Agent installation that incorporates all the drivers and SDK required to run, develop and/or recorded automated tests.
• A very intuitive in-browser recorder that can record tests on your physical devices and emulators in minutes, and doesn’t require APK/IPA/source code.
• Automatic Appium compatible test code generation from your recorded test.
• Built-in web based test execution reports & dashboard.
• Appium based SDK with extra capabilities for your coded tests. It currently supports Java, Python, JavaScript and C#. 
• Manage your entire team’s projects in one location (no git or 3rd parties are required).
• Quick deployment, able to run tests anywhere where a TestProject agent is running.
• Integrates with Sauce Labs and BrowserStack.
• Ability to run in Dockers.

TestProject Pros and Cons

TestProject Pros ✅	TestProject Cons 👎
Free	For C# .NET Core 2.1 is required
Supports Android/iOS on both Windows/macOS	While it easy for non-coders, configuring IDE/SDK environment can be challenging at first
No coding background needed
Background in app development not needed
One-click installation
Recorder can do complex actions, ever-expanding by community Addons
Easy to share tests
Not bounded to a local machine
Creates a comprehensive test report
Can run in Dockers
Built-in integration with Sauce Labs and BrowserStack
Easy for non-coders

 

Speed Tests of All Mobile Automation Testing Tools 

Below are two speed benchmarks for Android tests as well as iOS tests.
All times are in milliseconds – The lower the score, the better performance 😉

• XPath: Espresso does not natively support XPath.
• ID: For Espresso/UiAutomator = id/accesabilityId, on XCUITest = identifier/label.

Speed Tests UiAutomator vs. Espresso vs. Appium vs. TestProject 

 

Speed Tests XCUITest vs. Appium vs. TestProject 

We can conclude this speed benchmark with the following:

• As you can see in the charts above, TestProject is the fastest when it comes to Android tests, followed by all of the other proprietary frameworks.
• Appium (both for Android and iOS tests) seems to be trailing far behind, as well as TestProject for iOS.
• Espresso/UiAutomator/XCUITest all have a pretty tight matching score.
• As for the locators themselves, it seems that ID locators are slightly faster than XPath.

So, what is your favorite mobile automation testing tool? Share in the comments below! 👉