Top 5 iOS Testing Frameworks

4 November 2021 Stephan Petzl Leave a comment Test automation

Are you looking for iOS testing frameworks to improve your iOS testing strategy? An iOS testing framework can cut down time you’d otherwise have to invest to check apps for bugs manually. There are numerous ways to accomplish this, but you only have limited time and money. Even if you’re an expert at iOS testing, there’s always room for improvement. It’s beneficial to know which iOS testing frameworks to put into your toolbox.

In this article, you will learn about the most used frameworks for the automation of iOS testing.

Overview

• The top 5 iOS testing frameworks
• Which iOS testing framework is the best?
• iOS test automation – why is it important?
• Summary: what you really need to know about iOS testing frameworks

What are the Top 5 iOS testing frameworks?

1) Appium

Appium is a grown-up open-source test automation framework driven by a big community. Part of the reason could be its programming language variety and ability to test iOS, Android, and mobile web applications.

QAs can use any programming language they want to automate their application tests. Appium uses Selenium WebDriver APIs and language-specific client libraries to allow you to write tests in the language of your choice (Java, Python, JavaScript, PHP, etc). It also provides access to backend APIs and databases, which enables quite a lot of flexibility.

Pros of this iOS testing framework

• You can use any WebDriver-compatible language, such as Java, Objective-C, or JavaScript, to write tests in your language and tools of choice
• The framework as a whole is open source
• For unit testing, you can use the framework of your choice, such as XCTest
• Different ways of locating an element are supported (XPath, ID, Text…)

Cons of this iOS testing framework

• Setting up Appium including drivers is time-consuming (order of days)
• Appium’s open-source components are not always compatible with each other’s versions, maintenance of the test automation setup can consume significant resources
• Appium execution on iOS devices is relatively slow, which can backfire if tests are not executed regularly because of the big amount of time it takes to execute the whole test suite

Testing code example

driver.findElement(By.id("com.my.app:id/myComment")).click();
driver.findElement(By.id("com.my.app:id/myComment")).sendKeys("Hi there");
driver.findElement(By.name("Post")).click();

2) Repeato

Repeato is a low-code test iOS test framework that allows for automated testing. It was designed with a strong focus test stability and test engineer experience.

Repeato uses computer vision and machine learning to observe the UI and execute the interactions just as a normal user.

This is how it works: Tests can be recorded quickly, without the need to code. Afterward, tests can be edited and saved to a library.

Tests can be run individually or in batches on Mac, Windows, or also headlessly on a CI/CD server.

Pros of this iOS testing framework

• Extremely fast creation of teests
• Plug and play, all you need is Repeato Studio, which is set up within 5 minutes
• Can test iOS, Android, React-Native, Flutter, Ionic, native code
• Powerful features provided by its “Script Steps” (JavaScript or Command Line)

Cons of this iOS testing framework

• Not open-source, freemium model
• No device farm support (yet)

3) XCTest & XCUITest

Probably the most used iOS testing framework integrates smoothly with XCode’s testing workflow. It’s developed by Apple and therefore well supported and kept in sync with the iOS application framework.

Pros of this iOS testing framework

• XCTest is relatively simple to integrate into an app.
• Unit, UI, and performance testing are possible.
• Developers can utilize their existing abilities by using Objective-C and Swift to write tests.
• Xcode Server can execute tests automatically as a “bot” in the CI/CD process.

Cons of this iOS testing framework

• Testers need to have developer skills because  XCode is a complex piece of software and not everybody has the knowledge of being able to write code.
• Object-C and Swift are the only supported programming languages.
• Not available for Android

Testing code example

let app = XCUIApplication()
// look up element by id
let toggleButton = app.segmentedControls.buttons["toggleButton"]
// look up element by id
let dayLabel = app.staticTexts["Day: "]
let nightLabel = app.staticTexts["Night: "]
// check elements  
XCTAssertTrue(dayLabel.exists)
XCTAssertFalse(nightLabel.exists)
toggleButton.tap()
XCTAssertTrue(nightLabel.exists)
XCTAssertFalse(dayLabel.exists)

4) EarlGrey

EarlGrey is a Google-developed open-source native test framework for iOS UI test automation. Google uses this framework to test their native iOS apps such as Youtube, Google Calendar, Google Photos, Google Translate, etc. It provides increased synchronization features for stable UI testing to QA teams.

Enhanced synchronization options are available with the EarlGrey framework, EarlGrey synchronizes with the user interface, network requests, and multiple queues automatically. You can still manually implement specific timings if necessary.

Tests can be written in Objective C and Swift

Pros of this iOS testing framework

• Versatile framework with robust internal component synchronization
• The entire framework is open source.
• Integrates with XCode.
• Well maintained and under active development (5.3k ⭐️ & over 1200 pull requests on github)

Cons of this iOS testing framework

• Supporting some of the APIs may necessitate adding some additional code to your app.
• Setup is straight forward but can be challenging for users without developer experience
• Not available for Android
• A lot of unresolved issues (currently 180 open issues on github)

Sample code

import EarlGrey
import XCTest

@testable import EarlGreyExampleSwift

class EarlGreyExampleTest: XCTestCase {

  func testClickCommentBox() {
    // Select an button by accessibility ID and tap it
    EarlGrey.selectElement(with: grey_accessibilityID("PostComment"))
        .perform(grey_tap())
    // check if comment box went visible
    EarlGrey.selectElement(with: grey_accessibilityID("MyCommentBox"))
      .assert(grey_sufficientlyVisible())
    // Select element by text
    EarlGrey.selectElement(with: grey_text("Setting"))
      .perform(grey_tap())
  }
}

5) Detox

Detox (developed by Wix) is an end-to-end iOS testing framework that enables grey box testing to decrease the flakiness and unpredictability of black-box testing tools like Appium.

It can test apps, running them on the simulator, making it possible to observe the test runs. Another interesting detail is that  Detox is using Javascript, so you may use it with any of the many great JS test runner tools out there (Jest, Mocha, AVA,…).

Pros of this iOS testing framework

• Detox works on iOS and Android
• The tool’s flakiness reduces through advanced monitoring.
• Great support with continuous integration tools.
• Provides Javascript APIs that are straightforward to use.
• Open Source
• Well maintained by an active community (8.4k ⭐️ & over 1000 pull requests on github)

Cons of this iOS testing framework

• Detox isn’t nearly as well-known or supported as the testing frameworks mentioned above.
• It does not work on physical iOS devices (yet)

Sample code

describe('AllAppTests', () => {
  beforeEach(async () => {
    await device.launchApp({ newInstance: true });
  });

  it('should have login screen', async () => {
    await expect(element(by.text('Login'))).toBeVisible();
    await expect(element(by.text('Username'))).toBeVisible();
    await expect(element(by.text('Password'))).toBeVisible();
  });

  it('should show error message after tap', async () => {
    await element(by.text('Login')).tap();
    await expect(element(by.text('You need to provide a username!'))).toBeVisible();
  });

});

What is the best iOS testing framework?

To increase the confidence in your app, we recommend testing on different levels of your software stack.

Ideally, you would do unit testing, integration testing, and end-to-end or UI testing.

For apps with a backend, we recommend API testing.

But going full bulletproof is not always possible. You might decide to cover only one or two testing layers, in the end, it’s still a tradeoff between UX and costs, and each app developer team needs to find the sweet spot by themselves.

Anyway, this overview can help you get going:

• For unit testing, XCTest is the best option.
• For UI testing, XCUITest or Repeato are the best options.
• We didn’t cover API testing tools here, but we still want to mention ApiGee and SoapUI as possible options.

Why is it important to automate iOS tests with frameworks?

iOS test automation is critical when the goal is to create successful apps at a rapid pace.

A few advantages of automated testing of iOS apps are:

• Uncover bugs faster
• Get faster feedback when you automate testing for iOS apps
• Allow your team to stay ahead of schedule
• Save time when you release updates of your app frequently

Thus, using iOS test automation frameworks and tools is the only way to stay ahead of the competition to meet customer expectations in terms of stability and user experience.

Choosing which tests to automate

Have you thought about which use cases are at the core of your app? What are the most important use cases which must not break?

This is a great place to start creating a list of flows that could potentially be automated.

We wrote a little about what a strategy for test creation could look like.

Summary: Your takeaways on iOS testing frameworks

A well-thought-out iOS testing approach is critical to the development of a successful app. There are numerous iOS testing frameworks available today, but selecting the appropriate one for your purposes can be difficult.

With that in mind, your product and team will determine the ideal mobile testing tool selection.

Your testing frameworks should fit into your team rather than one that forces your team to work into it.

Like this article? there’s more where that came from.

• Mastering ADB Commands: Granting and Revoking App Permissions
• How to Lock and Unlock the Android Screen via ADB
• Locating Android Emulator Snapshot Files for Team Collaboration
• Automating Factory Resets on Android Devices via ADB
• How to Install ADB Without the Full Android SDK