There are several highly popular frameworks for mobile test automation, so I went on a mission to find the fastest mobile automation testing frameworks and best the locator strategies out there. Is there even a noticeable difference? 🤔
I created a short benchmark overview running similar tests across iOS and Android apps, while using different frameworks and locator strategies. For this overview I’ve created two simple apps:
- One is an Android app created with Android Studio on windows.
- One is an iOS app created with Xcode on a macOS.
For those of you who can’t wait until the end for the results – Here’s a recap of this speed benchmark:
- Appium and TestProject are the only 2 frameworks that support both iOS and Android.
- TestProject was by far the fastest one on Android (92-119 milliseconds), and is hassle-free compared to the others since you just need to install a lightweight Agent and go ahead running your tests, that simple.
- Appium takes time to set up, but is fairly easy to use when it comes to coded tests. However, it was by far the slowest on Android.
- Espresso/UiAutomator/XCUITest were all fairly close in terms of speed, and all support a similar set of features. Easy to use in their respective environments, however all use very basic recorders, which means they can only be utilized for coded tests apart from basic actions.
Table of Contents
- Benchmark Recap of all 5 Mobile Automation Testing Tools
- Mobile Automation Testing Tools – Summarizing Diagram
- Review of all 5 Mobile Automation Testing Tools
- Speed Tests of all 5 Mobile Automation Testing Tools
Mobile Automation Testing Tools – Summarizing Diagram
Review of all 5 Mobile Automation Testing Tools
Before diving into our speed benchmark, let’s review the 5 chosen mobile automation testing frameworks:
1. Appium
Appium is an open source tool for automating native, mobile web, and hybrid applications on iOS, Android and Windows platforms.
Appium has client libraries in Java, Ruby, Python, PHP, JavaScript and C#, which support Appium’s extensions to the Selenium WebDriver protocol. This makes Appium highly versatile, albeit with a few caveats:
- Installation can be a bit of a hassle. You need to either install all the separate packages and libraries on your own, or install node.js combined with something called NVM to help you manage the installation process.
- iOS tests require Xcode, which is Apple’s proprietary IDE, so you’d need to either use a macOS, or Windows with Appium Studio (community edition), or a macOS virtual machine (perhaps not entirely legal). If you are testing on both Android and iOS, this can be quite tedious unless you own a mac.
Appium uses the vendor-provided automation frameworks, which means its using XCUITest for iOS devices, and UiAutomator for Android devices.
Pros and Cons of Appium
Appium Pros ✅ | Appium Cons 👎 |
Free | Installation needs multiple steps, can be complex |
Supports both iOS and Android | Can run both iOS and Android only on macOS; Windows only supports Android |
Open source | Requires some technical skills |
Inspector session is very simple and intuitive | Requires programming skills |
If you like coded tests, it is convenient | Not easy for testing newbies |
2. XCUITest
XCUITest is Apple’s proprietary framework. It requires Xcode to run, so it means you can only use a mac to run it, or use the “bypasses” mentioned in the Appium review above, but you cannot run it natively on Windows! It supports Swift and Objective-C.
XCUITest Pros and Cons
XCUITest Pros ✅ | XCUITest Cons 👎 |
Convenient if you are familiar with iOS app development | Background in iOS app development required |
Xcode has a very intuitive recorder | Recorder can only support very basic actions |
Uses the same IDE you created your app on | Proprietary – You will need to learn the syntax if unfamiliar |
Directly supported/updated by Apple | Can be used mainly for unit testing |
Very good documentation and tutorials by Apple | This framework was built for developers rather than QA engineers |
Requires application source code/IPA |
3. UiAutomator
A Google unit testing framework. It supports Java and Kotlin, and you can run it on any Java supporting IDE.
UiAutomator Pros and Cons
UiAutomator Pros ✅ | UiAutomator Cons 👎 |
Convenient if you are familiar with Android app development | Background in Android app development required |
Has a very intuitive recorder | Recorder can only support basic actions |
Uses the same IDE you created your app on | Proprietary – You will need to learn the syntax if unfamiliar |
Directly supported/updated by Google | This framework was built for developers rather than QA engineers |
Very good documentation and tutorials by Google | Requires application source code/APK |
4. Espresso
This Android application testing framework might be useful for developers that follow the TDD (Test Driven Development) approach or develop their tests as part of the development lifecycle. The framework suggests a simple and small API, that is easy to learn and built on top of Android’s instrumentation framework.
Espresso Pros and Cons
Espresso Pros ✅ | Espresso Cons 👎 |
Frequent updates and provides enhanced support for newer Android versions | Need to build the APK before running tests |
Tests live in the app and supposed to run much faster | Application source code is required, meaning this framework is mainly for developers |
Provides Automatic Synchronization to make tests more stable | Waits for the main UI thread to be idle before test actions are performed |
It launches fast | Doesn’t support webviews. |
5. TestProject
TestProject is a next generation test automation platform, built on top of Appium and Selenium and aims to simplify the test automation experience. The biggest upside of TestProject is that it is one quick installation (that simplifies all of the Appium and selenium component installations and configuration), and you are good to go for testing on Android, iOS (even on windows!) and Web on any operating system (Windows, macOS and Linux). Here are some additional features that make TestProject perform better than others:
- Simple Agent installation that incorporates all the drivers and SDK required to run, develop and/or recorded automated tests.
- A very intuitive in-browser recorder that can record tests on your physical devices and emulators in minutes, and doesn’t require APK/IPA/source code.
- Automatic Appium compatible test code generation from your recorded test.
- Built-in web based test execution reports & dashboard.
- Appium based SDK with extra capabilities for your coded tests. It currently supports Java, Python, JavaScript and C#.
- Manage your entire team’s projects in one location (no git or 3rd parties are required).
- Quick deployment, able to run tests anywhere where a TestProject agent is running.
- Integrates with Sauce Labs and BrowserStack.
- Ability to run in Dockers.
TestProject Pros and Cons
TestProject Pros ✅ | TestProject Cons 👎 |
Free | For C# .NET Core 2.1 is required |
Supports Android/iOS on both Windows/macOS | While it easy for non-coders, configuring IDE/SDK environment can be challenging at first |
No coding background needed | |
Background in app development not needed | |
One-click installation | |
Recorder can do complex actions, ever-expanding by community Addons | |
Easy to share tests | |
Not bounded to a local machine | |
Creates a comprehensive test report | |
Can run in Dockers | |
Built-in integration with Sauce Labs and BrowserStack | |
Easy for non-coders |
Speed Tests of All Mobile Automation Testing Tools
Below are two speed benchmarks for Android tests as well as iOS tests.
All times are in milliseconds – The lower the score, the better performance 😉
- XPath: Espresso does not natively support XPath.
- ID: For Espresso/UiAutomator = id/accesabilityId, on XCUITest = identifier/label.
Speed Tests UiAutomator vs. Espresso vs. Appium vs. TestProject
Speed Tests XCUITest vs. Appium vs. TestProject
We can conclude this speed benchmark with the following:
- As you can see in the charts above, TestProject is the fastest when it comes to Android tests, followed by all of the other proprietary frameworks.
- Appium (both for Android and iOS tests) seems to be trailing far behind, as well as TestProject for iOS.
- Espresso/UiAutomator/XCUITest all have a pretty tight matching score.
- As for the locators themselves, it seems that ID locators are slightly faster than XPath.
Well, as far as I am concerned, I am sold on Test Project to consider it for Mobile Automation for apps. Kudos!