A Complete Guide to iOS Testing Frameworks

The need for more robust systems in the development and testing domain is affected directly by one or more of the following things – the number of applications flooding the app stores, the number of devices running the iOS in the market, and the difficulty in testing these applications (a direct consequence of complexity).

As in our case today, application and devices metric is so high that manual real device testing cannot be imagined. Thankfully, there are efficient and feature-rich iOS testing frameworks in the market that make iOS application testing faster, easier, and sometimes without demanding any resources from us 😃

In this post, we will see 8 frameworks that are helping testers all over the world deliver high-quality iOS applications to their users.

Table of Contents – iOS Testing Frameworks – A Complete Guide

• Do we really need iOS testing frameworks?
• What are iOS testing frameworks?
• Importance of using iOS testing frameworks
• Top iOS testing frameworks
  • Appium
  • TestProject
  • EarlGrey
  • XCTest / XCUITest
  • Detox
  • Calabash
  • OCMock
  • KIF
• iOS testing frameworks at a glance
• Conclusion

Do we really need iOS testing frameworks?

In 2016, Apple sold its billionth mobile phone (iPhone) in the market. By January 2019, 900 million people were actively using iPhones in their daily lives. Two years later, that number became 1.65 billion. That’s huge! And this number is only increasing year after year.

On top of it, The numbers are only for iPhones. iOS running devices also include iPads (before iPadOS), and there are more than 500 million floating in the market. These devices are unique in every way I can imagine. Their specifications are different, processor power is different, they even don’t have the same screen size which can be a major problem while running the iOS applications.

Going ahead manually to test the iOS application can waste a lot of time, as each of the devices will present its own challenges, and coping with everything including the APIs will just delay everything repeatedly ⏳

The solution to this huge fragmentation is iOS testing frameworks, which are designed specifically to be incoherent to the iOS code and facilitate testing on these devices.

What are iOS testing frameworks?

A testing framework is a tool to help you conduct test executions efficiently. As a subset, an iOS testing framework is a testing framework developed especially for iOS mobile application testing 📲

iOS testing frameworks also provide additional features in a single place such as using Apple’s API, analyzing the test targets, collaborating with other testers, generating screenshots, organizing project files, and generating extensive detailed reports which are not possible during manual testing. The features and highlights of various iOS testing frameworks will enlighten you more in the later sections. 

Importance of using iOS testing frameworks

So, what if I say that I am ready to test my iOS application manually without any help? What parameters am I not considering?

• Expense: iPhones are already priced on the upper bar of the mobile market average. Purchasing all the target devices or taking them on the lease would be a huge expense and can eat up a significant amount of your funds if you are an individual or a small team startup.
• Time wastage: Personally, I have always felt that automation testing of iOS applications always saves my time. The frameworks help in test creation and remind me of the test guidelines and rules to follow, provide ready-made snippets, and even correct a few things (such as URL encoding) behind the scenes.
• Maintenance overhead: Maintenance of an application is always something we need to ponder beforehand during the first release. Applications need to be updated, modified, and sometimes redesigned completely based on newer technologies and frameworks. All these things will require changing scripts and constructing newer ones. Maintenance overhead can be reduced to a large extent if the tester goes for frameworks.
• Analytics: Using a framework brings out detailed analytics with respect to the tests you are performing. These analytics are very important and helpful in determining the test quality of the process. For example, bringing out the test coverage metric can help us understand how much code we are considering in testing. If it’s too low, the test quality is poor and we can work on it.
• Resource utilization: Using a manual approach requires us to install everything on our personal machine. Along with testing software, our personal machine also runs other programs, and therefore resource management becomes another hectic task. iOS testing frameworks take little space and set up everything on their own, which is the best choice for resource management at a very low cost.

Now that we are convinced about using the iOS testing frameworks in our next iOS project, let’s look at the different options we have 🔍

Top iOS testing frameworks

The following section details the top iOS testing frameworks that are popular among iOS testers currently:

1) Appium

Appium is an extremely popular open source test automation framework among the testing community. The part reason could be its versatility with programming languages and the ability to test iOS as well as Android and mobile web applications 🌐

As Appium’s official website mentions, “Appium is built on an idea that testing native apps should not require installing SDK”. With Appium, the user gets full flexibility with the tools and framework in which a tester is proficient and confident. 

iOS testing in Appium is performed using JSONWireProtocol and Selenium WebDriver. For all the Selenium fans, it becomes extremely easy to test an iOS app using Appium.

Pros

• Flexibility in choosing the framework of your choice.
• Since WebDriver is used, Appium supports a long list of programming languages.
• Open source framework and hence boasts a large community.
• Works with native apps on Android, iOS as well as mobile web testing.
• Similar APIs can be used for automation tests for both the major platforms – Android and iOS.
• Installing other heavy tools is not required while working with Appium.
• Recompilation of the application code is not required in Appium.
• Appium can work along with emulators, simulators, or real devices.
• Brings a good integration facility with CI/CD tools.

Cons

• The capability to locate images on an iOS mobile application is not supported.
• Open source components installed in Appium should have compatible versions with each other.
• Testers often get confused while configuring Appium on their systems.
• Appium execution while working with iOS devices is very slow and can be annoying for the testers.

Code example:

text_field = driver.find_element_by_id('TextField1')
 assert text_field.get_attribute('value') is None
 text_field.send_keys('Hello World!')
 assert 'Hello World!' == text_field.get_attribute('value')

Bottom line 

Appium is a great software and is popular in the community. It has been used for quite a while now and enjoys good support. It is open source and can perform the majority of the tasks required as a tester.

Appium however fails to satisfy those people who need things to move faster and don’t want to get stuck in configuring, setting up or managing components. It can also slow down testing when the device matrix is large.

2) TestProject

TestProject iOS testing framework is based on Appium (as well as Selenium), and is pretty much a framework for everyone (even for Android, Web and API testing). Whether you want a specific operating system or don’t want to use XCode, TestProject has got you covered with a simple test framework with just two pre-requisites: TestProject agent and an Apple developer account for physical devices, or macOS and Xcode for Simulators 🥇

Pros

• Easier to learn and quicker to execute tests.
• Comes with iOS devices and iOS simulator support.
• Requires minimum configurational setup so things can start quickly.
• Includes TestProject recorder equipped with element locator and AI technology for efficient testing.
• The operating system is independent and can be run on Windows operating system too.
• Automation is supported within XCode simulators on MacOS.
• Supports all popular programming languages with open source SDK for- Python, Java, JavaScript & C#

Cons

• Documentation is sometimes released late. A tester might need to go explore the tool himself or reach out to their customer support.
• When it comes to developing addons it requires coding skills and if you have none it might be challenging.

Bottom line

TestProject is an awesome free framework and has garnered positive reviews from all around the world. It is easy to understand and takes no time in setting up the configurations.

The best TestProject advantage is that it does not force the testers to use any specific platform or language. If you are interested in writing the code yourself, it supports all popular programming languages- Python, Java, JavaScript & C#. The flexibility of the framework makes it a must-try iOS testing framework.

3) EarlGrey

EarlGrey is an open source iOS testing framework developed by Google to test their native iOS applications which include YouTube, Google Calendar, etc 🍵 EarlGrey 1.0 is now officially deprecated and version 2.0 is recommended for the iOS testers.

With EarlGrey 2.0, the testers can combine their EarlGrey tests with XCUITest which is Apple’s official testing framework. Also, as stated in Google’s official documentation, “EarlGrey 2.0 uses UI testing target and not unit testing as a target like in EarlGrey 1.0”.

Pros

• EarlGrey’s synchronization features are excellent and can synchronize automatically with the network and UI components. 
• The testers can also set their own timing if required.
• EarlGrey improves the test’s stability as the actions are performed after the UI settles into a steady state.
• EarlGrey’s tests can be run directly from XCode or through the command line.
• Open source in nature.

Cons

• May require additional coding in the application to support some of the APIs.
• Google fetches the tests executions and other test analytics from the testers. This can be turned off but not all of the testers are aware of it.

Code example

var firstMatch = true
  let matches: MatchesBlock = { (element: AnyObject!) -> Bool in
    if firstMatch {
      firstMatch = false
      return true
    }
    return false
  }

Source: EarlGrey examples from the official repo.

Bottom line

EarlGrey is a great testing framework for iOS applications. However, EarlGrey does blend very easily with Google applications and not so easily with other native applications. It has a shallow learning curve and its strength comes from its mingling powers with UI test elements, XCode and XCUITest.

4) XCTest / XCUITest

XCTest is Apple’s iOS testing framework that’s fully compatible with XCode (and is released in parallel to XCode). XCTest is very versatile and can perform tests on various levels such as unit tests, performance tests and user interface tests ✅ XCUITest can be considered as a sub-class of XCTest and works specifically towards the user interface of the application. 

Pros

• XCTest provides a single solution to a variety of tests.
• For iOS app developers, the testing becomes very easy due to the same language.
• The test execution is faster than other iOS testing frameworks.
• XCTest has a shallow learning curve and can be grasped easily given that the tester is familiar with Objective-C/Swift.
• XCUITest provides UI recording capabilities and generating code out from it.
• Easier to deal with CI/CD tools and get seamless integration.

Cons

• Limited language support. Unfamiliar testers might have to go through a learning phase which can take time.
• XCTest/XCUITest does not have cross-platform support.
• Only technical testers with a programming background can perform testing on XCTest.
• Deeplinking is not supported by XCTest/XCUITest.

Code example

class TableValidationTests: XCTestCase {
    /// Tests that a new table instance has zero rows and columns.
    func testEmptyTableRowAndColumnCount() {
        let table = Table()
        XCTAssertEqual(table.rowCount, 0, "Row count was not zero.")
        XCTAssertEqual(table.columnCount, 0, "Column count was not zero.")
    }
}

Source: Official XCTest documentation

Bottom line

XCTest/XCUITest is similar to what everything has been with Apple. It is a closed iOS testing framework with programming limitations. This is a great disadvantage with people familiar with testing iOS apps on cross-platform frameworks with the language of their choice. Honestly, XCTest is perfect for those who have developed iOS applications and work with XCode for their development purposes.

5) Detox

Detox is an end-to-end testing tool that uses gray box testing to control the flakiness and unpredictable behavior of black-box testing tools such as Appium 🤖

Detox comes with two very strong USPs. First, it can test an application while it is running on the simulator and therefore works as if a real user is finding the defects. Second, Detox is Javascript based and therefore you can run anything that supports it.

Pros

• Detox comes with cross-platform support.
• Can test an application just like a real user.
• Advanced monitoring reduces the flakiness of the tool.
• Very high and smooth support for continuous integration tools.
• Does not depend on Webdriver for testing support.
• Uses very simple and easy-to-use APIs written in Javascript.
• Open source with a lot of contributions since its inception.

Cons

• Testing on real iOS devices is still not supported.

Code example

describe('Login flow', () => {  
  it('should login successfully', async () => {
    await device.reloadReactNative();    

    await element(by.id('email')).typeText('[email protected]');
    await element(by.id('password')).typeText('123456');
    await element(by.text('Login')).tap();      

    await expect(element(by.text('Welcome'))).toBeVisible();
    await expect(element(by.id('email'))).toNotExist();
  });  

});

Source: Official Detox Repo

Bottom line: Detox brings important things to the table. It is gray box testing and performs end-to-end testing using Javascript which is one of the most popular languages in the world today. So, thumbs up for that! The only problem I find with Detox is that it does not support iOS real device testing and uses a simulator to achieve the same.

Using a simulator is also good but cannot check the quality of the application better than a real device. Therefore, I think this factor can be a deal-breaker for many testers.

6) Calabash

Calabash is a big name among the iOS testing frameworks listings and communities because of its stability and approach towards the construction of the tests. Calabash uses Cucumber, which enables non-tech people to constructs tests easily 💫 Calabash comes with a long list of language support and can run on iOS as well as Android applications.

Pros

• Open source and one of the major iOS testing frameworks in focus. Hence, continuous development.
• Supports Python, C#, PHP, Ruby, Perl, and many more.
• Constructs tests with the BDD approach which is extremely helpful for non-tech people.
• It’s a cross-platform testing framework and hence enables tests across platforms without creating newer ones.
• Supports iOS emulators as well as physical devices.
• Calabash is known to be a very stable iOS testing framework.

Cons

• Calabash requires coding outside its BDD bubble.
• Takes time to run the application.
• Does not have a very large community to help.

Code example

Scenario: Money Transfer Changes Balance in Wallet
Given that I have $100 in my wallet
And I add $200 to my wallet
My wallet balance increases to $300

Bottom line

Calabash has gained a lot of popularity, and the main reason behind this is the support for the BDD approach.

BDD is always preferred by people as it can get the task done without learning the programming languages and creating complex logic in it. But people with programming backgrounds often criticize Calabash for the same reason. Calabash also does not have a large community compared to Appium.

7) OCMock

OCMock is an open source project enhancing the iOS testing methods by creating mock objects. OCMock stands for Objective-C Mock which means that the tester can create the mock objects using Objective-C language 👩‍💻

OCMock is actively developed and includes three types of mocks – stubs, dynamic mocks, and partial mocks. The examples and definitions can be learned at the OCMock official website.

Pros

• Easy learning curve and easy to create objects of various types.
• Extensive documentation to help the testers.
• iOS app development and OCMock both use Objective-C and therefore it becomes easier to work with OCMock.
• It’s an open source framework and therefore sees an active development from people all around the world.

Cons

• Just to mock objects. Does not provide a complete iOS testing solution.

Code example

id userDefaultsMock = OCMClassMock([NSUserDefaults class]);
// set it up to return a specific value when stringForKey: is called
OCMStub([userDefaultsMock stringForKey:@"MyAppURLKey"]).andReturn(@"http://testurl");
// set it up to return the specified value no matter how the method is invoked
OCMStub([userDefaultsMock stringForKey:[OCMArg any]]).andReturn(@"http://testurl");

Source: Official OCMock Documentation

Bottom Line

OCMock is not a testing framework but works with many. In its documentation, OCMock does not mention anywhere about being a “testing framework” or “testing solution”. Although, OCMock does mention enhancing the abilities of XCTest and other such frameworks that support it.

So, if you are someone who already knows an iOS testing framework and wants to enhance it by adding OCMock, it is actually a great choice and helps a lot in creating mock objects. But if you are expecting a complete testing solution with OCMock, you will be disappointed.

8) KIF

KIF is an integration testing framework designed for iOS applications. KIF stands for “keep it functional” and is used for UI testing on an iOS application. KIF uses the XCTest target to build the automation tests. Although, it does use undocumented APIs and therefore should never be used in production ❌

Also, KIF mentions that it needs to be added to the unit testing target and not the UI testing target which can be a bit confusing as it is used for UI testing.

Pros

• Objective-C support and therefore becomes easy for the developers.
• KIF testing is somewhat similar to mimicking a user’s actions which is helpful.
• KIF does not require an additional configurational setup.
• XCTest advantage gives it the upper edge towards easy integration with the XCode tools.
• KIFs tests are noted to be faster and quicker.

Cons

• Language constraint as only Objective-C is used in testing.
• KIF does not enjoy great support as of now.
• KIF does not support popular frameworks such as Cucumber for which it has been criticized.

Code example

@implementation LoginTests

- (void)beforeEach
{
    [tester navigateToLoginPage];
}

- (void)afterEach
{
    [tester returnToLoggedOutHomeScreen];
}

- (void)testSuccessfulLogin
{
    [tester enterText:@"[email protected]" intoViewWithAccessibilityLabel:@"Login User Name"];
    [tester enterText:@"thisismypassword" intoViewWithAccessibilityLabel:@"Login Password"];
    [tester tapViewWithAccessibilityLabel:@"Log In"];

    // Verify that the login succeeded
    [tester waitForTappableViewWithAccessibilityLabel:@"Welcome"];
}

@end

Source: Official KIF repo

iOS testing frameworks at a glance

This has been a long list, and I am sure recollecting all the points would be hard 😅 In this section below, I have organized these iOS testing frameworks into tabular form to let you compare everything in a single table: 

W – Windows

M – MacOS

L – LinuxOS

Conclusion

Automation testing has started to dominate the testing market by its near-zero error margins and faster test execution capabilities. But in general, automation is just a wider umbrella that covers a lot of frameworks that have their own set of qualities, strengths, and obviously their weak links.

Although, I believe the weakness of software is not generic but specific to specific users. If my automation tester is highly proficient in Objective-C, I can happily choose an Objective-C-supported iOS testing framework (even if it is the only language supported!).

In this post, we tried to split out the differences keeping the comfort of the testers in mind. For example, a framework supporting five languages covers more testers under its radar than a framework supporting two. I hope these differences will help you select the best tool according to your preferences and requirements.

I hope this article provides clarity on these mentioned iOS testing frameworks. let me know if you have any questions 🤗 Till then, happy testing!