Learn the basics of Android UI testing & frameworks

20 March 2020 stoefln Leave a comment Test automation

An Android UI testing framework and process is a crucial part of app development and necessary to produce high-quality applications in due time.

There are many such frameworks available. However, each framework has advantages and disadvantages: not all frameworks are equally effective or inexpensive. Some can cost you quite a bit. Choosing the right Android UI testing framework can be a challenge.



What are the benefits of Android UI testing?

Android testing summarizes several facets of testing, for example interface testing and performance testing. In this article we’ll focus on UI testing.

Testing the user interface of the Android apps you ship is a crucial element of the development process that should not be skipped. It makes sure the app works as expected, is user friendly and without bugs. Only through extensive testing you can be sure that users will use your application successfully and enjoy the app.

General benefits of UI testing are:

  • The app will have less bugs
  • Users enjoy better usability
  • Easier use results in higher user satisfaction
  • Users will issue less complaints
  • Higher user loyalty
  • The app/company is more competitive

What is Android test automation?

First of all, let’s clarify a basic question: What exactly is automated testing?

There are two main types of testing: manual and automated.

  • In manual testing, you have to execute test cases by yourself without the use of tools or scripts.
  • In automated testing, however, you will use testing tools and scripts.

What are the benefits of Android test automation?

There are many reasons why automated Android testing is a great thing to do:

  • Automated tests are much faster than manual tests.
  • When test cases are run many times and over longer periods of time, automated testing becomes more practical.
  • More reliable for tests that have to be done repeatedly since the process is done the same way each time.
  • Reliability means that there are less chances of inaccuracy due to human error.

Thus, testing the UI of Android apps is a process that can be automated in big parts; automating Android UI testing has many advantages, for example:

  • Saves time
  • Requires less developer energy
  • It saves money
  • Assures test reliability
  • Assures testing processes

Thus, automated testing Android apps comes with many benefits.

When to automate Android testing instead of manual testing?

Automating Android testing can be a useful addition to manual testing and make your testing process more holistic.

Automated testing for Android apps usually turns out to be extremely useful in these cases:

  • In case a task needs to be executed repeatedly.
  • When doing Regression Testing: since the code may need to be changed frequently.
  • In case of Performance Testing: where we want to test how an app would behave while being used by thousands of users.
  • When doing Load Testing: where we want to test how the app handles different stages of load such as data traffic, location and slower mobile network speeds.

What are the best Android UI testing frameworks?

It might seem difficult to get a good overview of the Android test frameworks out there. There is an obvious reason for the huge amount of Android UI testing frameworks: They all have their specific niche. Thus. You might need a set of frameworks because there is no jack of all trades testing solution.

There’s a huge number of tools that allow for Android UI testing, for example:

  • Appium
  • Robotium
  • Repeato
  • Calabash
  • Selendroid
  • UI Automator
  • Espresso
  • Monkey Runner

Which tool is the best for your project is difficult to tell – there are many factors that influence the decision.

To make choosing the best solution much easier for you, we have compiled a list of the eight Android test automation tools that you can use to carry out automated testing on your Android applications.

1. Repeato

Repeato is a tool for Android test automation which allows you to create tests via record & play – thus, the test author does not have to have coding skills. Repeato works based on computer vision and machine learning to recognize and interact with the elements in your user interface.

Thus, testing becomes feasible for people without coding skills; also, thanks to computer vision, tests are very robust. When needed, tests can be recreated quickly.

Repeato features

  • Creating tests via a test recorder only takes few minutes
  • Blackbox testing (no code required)
  • Testing native widgets (GPU rendered things like maps or 3D content)

Repeato is free for small projects, check the Android test automation tool out and simply try it out.

2. Appium

Appium is probably the most popular automated testing tool for Android that there is. It is an open-source tool. It’s free and supports a wide range of programming languages. These include Java, Java-Script, Objective-C, Ruby, Python, and many more.

Although Appium is a cross-platform tool, we will be discussing its applicability with Android here. It uses either UIAutomater or Instrumentation for Android, and these frameworks are used under Selenium WebDriver, which gives developers the luxury of being able to use several languages.

Furthermore, Appium allows you to test many types of mobile applications. This includes Native, Hybrid, and Mobile Web. Lastly, Sauce Labs supports Appium for testing with real devices, simulators, and emulators.

Therefore, Appium is an extremely versatile platform for automated Android testing.

Appium features

  • No installation required.
  • A vast and active community.
  • The app does not need to be modified to be tested using Appium.
  • Appium UI Inspector: a tool that comes with Appium to assess an app’s UI components.


3. Robotium

Robotium is an open-source tool for Android test automation. It was released back in 2010 and by now has become quite capable and reliable. Since then, its test execution and test readability speed have increased a lot.

Robotium makes its tests more robust by using run-time binding to UI components. It comes as a jar file that you have to compile with the app that you are testing. Not only does it support both Native and Hybrid app testing, but it also supports the basic Android UI tools such as activities, menus, and buttons as well as gestures.

Robotium uses Java as its language of choice. Using Robotium, developers can write functional tests, system tests as well as acceptance test scenarios that cover a host of Android activities. You can also create Automatic GUI (Graphical User Interface) tests.

Similar to Appium, Robotium is supported by Sauce Labs, which allows for testing on real devices.

Robotium features

  • Has device control functions such as taking screenshots and changing screen orientation.
  • Can run tests on emulators and real devices.
  • Supports multiple testing at the same time.
  • Robotium recorder: A paid plugin to speed up the testing process.

4. Calabash

Calabash is another open-source automated testing framework for Android that works for both Native and Hybrid apps. It allows you and even people without coding skills to conduct automated acceptance tests on apps. Xamarin created it.

Calabash supports many languages, including Java, Flex, Ruby, and .NET. It has libraries that enable test scripts to interact with Native and Hybrid applications. You can configure your Calabash tests to run on many different Android devices smoothly. They provide real-time feedback and validation.

Furthermore, Calabash works for Behavior-Driven Development (BDD). This is slightly different and more advanced than Test-Driven Development. Essentially, in BDD, tasks are created in which the intended behavior of the app is specified. The idea behind BDD is that the software is built right from the vision of the owner without the developer having to interpret how it should behave.

Calabash features

  • Configuration facility to make tests compatible with a wide range of devices.
  • Grants access to the Xamarin test cloud.
  • Supports Cucumber (a testing tool for BDD).
  • Cucumber Automation lets you write tests in simple English.
  • Enables automatic UI interactions within an application: pressing buttons, entering text, etc.

5. Selendroid

Selendroid is another UI test automation tool. It is used for testing Android Native as well as Hybrid applications. It tests apps using the Android driver WebView app. Tests have to be written using the Selenium 2 client API.

You do not need to modify your app while using Selendroid. You can also extend it using your own extensions during run-time. It can interact with multiple android devices, including emulators, at the same time. This feature is also owed to Selendroid’s distinct feature that you can plug and unplug devices without interrupting the test.

Using Selendroid, you can also execute parallel test cases on multiple devices. It contains an Inspector tool that can analyze the UI elements of the app that you are testing. Unlike other tools like Appium, Selendroid can find UI elements for older versions of Android, too, as early as Android 4.4 KitKat.

Selendroid features

  • Hot plugging.
  • Compatibility with JSON Wire Protocol.
  • Advanced User Interactions API: Support for gestures.
  • Emulators are started up automatically.
  • Supports different locator types.
  • Supports many versions of android from Android 4.0 KitKat onwards.
  • The built-in Inspector simplifies test case development.

Watch a Selendroid mobile web testing demo

6. UI Automator

UI Automator is another Android test automation tool that allows for UI testing of Android applications. It is developed and maintained by Google. It supports only Native Android apps.

UI Automator tests are written in Java. Moreover, it is a black-box testing app. This essentially means that you do not need to know the internal structure of the app being tested and can rely solely on the visible UI elements.

It gives you the ability to switch between installed and system apps, enables cross-app functional testing, and lets you test multiple apps. It comes with two sets of APIs; one allows it to manipulate the app’s UI components, while the other enables it to perform various tasks on the device.

UI Automator features

  • UI Automator Viewer: GUI tool to scan current UI components on the screen.
  • Performs interactions on system apps and user apps.
  • Performs actions such as opening app launcher in test device.
  • Simulate buttons on the screen, such as the home button.

7. Espresso

Espresso is also a useful testing tool for the automated testing of Android UI apps. App developers and testers widely use it. This solution is also from Google. Its tests are quick to execute and reliable.

On the other hand it’s not trivial to create tests. You need a lot of tweaks to really to get them stable. We covered some of the issues of espresso, i.e. about idling resources and thread sleep.

Espresso uses white-box testing, which is responsible for its fast speed. This means that you must be well-aware of the internal structure of the application while using Espresso to test it. Tests have to be written in Java – as with most Google-maintained frameworks.

Furthermore, it’s an open-source framework; hence you can easily extend it to better suit your needs. Another benefit of Espresso is that it allows you to write user interface tests for your application.

You can even create and record a scenario of interacting with the device. Capture snapshots to confirm the behavior of UI elements of the app. Espresso will use the recording and generate UI test cases for your app.

Espresso features

  • Tests can be built-in APK form, which can be installed and used quickly.
  • Does not communicate with a server.
  • Provides very stable tests thanks to the synchronized method of executing tests.
  • Works flawlessly with Android Studio IDE.

Read more about Android testing with Espresso, or about the differences between Espresso vs Robotium.

8. MonkeyRunner

MonkeyRunner is a popular tool for running automated functional tests on Android applications. The tests are written in Python.

MonkeyRunner gives you multiple device control. You can run tests across multiple devices or emulators at once, even by physically attaching them. It was developed to test applications at functional level, but you can use it to do more than just that.

This tool can also carry out something known as Regression Testing. This method of testing involves running an application and comparing the output to an already known set of correct outputs.

It also comes with an API that allows it to control functions of the device. Since MonkeyRunner uses Python, you can create modules and programs to control Android devices. Furthermore, you can extend MonkeyRunner with plugins and therefore expand its usability.

MonkeyRunner features

  • Multiple-device support.
  • Functional testing.
  • Tests can even be written using a recording tool.
  • Supports Regression Testing.

As you see, each tool has its own advantages. To make the choice easier for you, here’s an infographic that can also help you decide on some important factors.

This image helps you to get a better picture of the current situation and choose the Android test framework that is the best for your project:

Choosing the best android UI testing tool

You can find more information about the factors to consider when opting for a UI testing framework and a detailed description of Android testing frameworks in our other articles.

How much does Android UI testing automation cost?

The costs of Android UI will obviously depend on the framework you chose.

Here is some overview which might help you to calculate your cost if you want to know the cost of a full fledged device cloud setup:


ToolCosts for setupCosts per monthTech know-how neededComplexityIntegrations
Jenkins (self hosted)– 3 workdays for initial test creation*

– 2 workdays jenkins installation

– 5 workdays testing integration

– 1 workday server administration

– Cost for hosting

  • scripting tests
  • setting up a server environment
  • integrating tests into CI
  • delivering test results to team (slack / email)
firebase testlabInsert Custom HTML

– 3 workdays for initial test creation*

– half a workday for cloud service setup

$1 per hour per virtual device

$5 per hour per real device

scripting tests (kotlin/java)mediummedium
Experitest– 3 workdays for initial test creation*

– half a workday for cloud service setup

starting from €199*scripting tests (kotlin/java/c#/python/…)lowhigh
Bitbar– 3 workdays for initial test creation*

– half a workday for cloud service setup

starting from €159*scripting tests (kotlin/java/c#/python/…)lowhigh
Saucelabs– 3 workdays for initial test creation*

– half a workday for cloud service setup

starting from €298*scripting tests (kotlin/java/c#/python/…)mediumhigh
Google App Crawler– 1 workday for inital test setup*$1 per hour per virtual device

$5 per hour per real device

Repeato– 1 workday for initial test creation*starting from $29*lowmedium

* To make the Android UI testing frameworks comparable, we made the following assumptions:

  • Price per month
  • Test automation support
  • Unlimited testing units
  • 1 User access
  • Real devices & emulators
  • an initial workload for the creation of 10 UI tests


In a nutshell: what you need to know about Android UI testing

UI testing is a crucial development step that often gets not enough attention because it seems painful at first. However, if you are aware of the testing requirements and can build the testing processes accordingly, you’ll make sure to ship flawless apps. There are many There are numerous frameworks available to assist with Android UI testing; you simply need to select the ones that best suit the needs of your project.

Tags: , , ,

Like this article? there’s more where that came from.

Leave a Reply