20 March 2020 Leave a comment Test automation, Tools & Frameworks
Ensuring that your app is bug-free and responsive across different devices and platforms can be challenging. This is where automated testing frameworks come in handy. In this article, we’ll explore the top Android test automation frameworks, their features and limitations, and how they compare to one another, helping you select the best framework for your project needs.
In this article we look at:
- Why to automate Android testing
- The best test automation frameworks in 2023
- How to choose the best Android test automation framework:
- Do I need to have Continous Integration support?
- Do I need to pay for a device farm to test my app?
- What if I don’t have access to the source code of the app under test?
- Do I want to test native (C++, OpenGL) content in my android app?
- Do I have enough developer resources allocated to set up my testing environment?
- Do I need to test a multi-app user flow?
- Do I want to test pixel perfect rendering?
- Which ones are the right UI testing tools for my Android app then?
- How much does a UI testing cost?
Why to automate testing on Android
The importance of Android automation testing cannot be overstated. Decent testing ensures that an app’s user interface functions correctly and is responsive across different devices and platforms. With a well-functioning UI, users are more likely to have a positive experience with the app, leading to increased engagement, retention, and revenue.
Additionally, detecting and fixing bugs early in the development cycle saves time and resources in the long run.
Thus, the questions of manual vs. automated testing pops up. Automated testing frameworks optimize the testing process, making it faster and more efficient, allowing developers to focus on other aspects of the app’s development.
Ultimately, incorporating automated testing into the development process is critical to delivering a high-quality Android app that meets user expectations and drives business success.
The best Android test automation frameworks
To make choosing the best solution for Android automation testing much easier for you, we have compiled a list of eight commonly used tools; scroll down for in-depth information on each of these tools.
- UI Automator
- Monkey Runner
By the way, you might have also heard about TestProject.io which used to be pretty famous in the area – that tool has been shut down late 2022.
Repeato is a no-code testing tool that allows you to create tests via record & play – thus, the test author does not need coding skills. It’s ideal for automating testing processes that you would otherwise need to manually repeat.
Repeato works based on computer vision and machine learning to recognize and interact with the elements in your user interface.
Thus, testing becomes feasible for people without coding skills; also, thanks to computer vision, tests are very robust. When needed, tests can be recreated quickly.
Repeato features and capabilities
- Creating tests via a test recorder only takes few minutes
- Blackbox testing (no code required)
- Testing native widgets (GPU rendered things like maps or 3D content)
- Automatically check memory usage
Repeato is free for small projects, have a look and simply try it out.
Appium is probably the most popular Android automation testing tool that there is. It is an open-source tool. It’s free and supports a wide range of programming languages. These include Java, Java-Script, Objective-C, Ruby, Python, and many more.
Although Appium is a cross-platform tool, we will be discussing its applicability with Android here. It uses either UIAutomater or Instrumentation for Android, and these frameworks are used under Selenium WebDriver, which gives developers the luxury of being able to use several languages.
Furthermore, Appium allows you to test many types of mobile applications. This includes Native, Hybrid, and Mobile Web. Lastly, Sauce Labs supports Appium for testing with real devices, simulators, and emulators.
Therefore, Appium is an extremely versatile platform for Android testing.
Appium features and capabilities
- No installation required.
- A vast and active community.
- The app does not need to be modified to be tested using Appium.
- Appium UI Inspector: a tool that comes with Appium to assess an app’s UI components.
Robotium is an open-source tool for Android test automation. It was released back in 2010 and by now has become quite capable and reliable. Since then, its test execution and test readability speed have increased a lot.
Robotium makes its tests more robust by using run-time binding to UI components. It comes as a jar file that you have to compile with the app that you are testing. Not only does it support both Native and Hybrid app testing, but it also supports the basic Android UI tools such as activities, menus, and buttons as well as gestures.
Robotium uses Java as its language of choice. Using Robotium, developers can write functional tests, system tests as well as acceptance test scenarios that cover a host of Android activities. You can also create Automatic GUI (Graphical User Interface) tests.
Similar to Appium, Robotium is supported by Sauce Labs, which allows for testing on real devices.
Robotium features and capabilities
- Has device control functions such as taking screenshots and changing screen orientation.
- Can run tests on emulators and real devices.
- Supports multiple testing at the same time.
- Robotium recorder: A paid plugin to speed up the testing process.
Calabash is another open-source testing framework for Android that works for both Native and Hybrid apps. It allows you and even people without coding skills to conduct automated acceptance tests on apps. Xamarin created it.
Calabash supports many languages, including Java, Flex, Ruby, and .NET. It has libraries that enable test scripts to interact with Native and Hybrid applications. You can configure your Calabash tests to run on many different Android devices smoothly. They provide real-time feedback and validation.
Furthermore, Calabash works for Behavior-Driven Development (BDD). This is slightly different and more advanced than Test-Driven Development. Essentially, in BDD, tasks are created in which the intended behavior of the app is specified. The idea behind BDD is that the software is built right from the vision of the owner without the developer having to interpret how it should behave.
Calabash features and capabilities
- Configuration facility to make tests compatible with a wide range of devices.
- Grants access to the Xamarin test cloud.
- Supports Cucumber (a testing tool for BDD).
- Cucumber Automation lets you write tests in simple English.
- Enables automatic UI interactions within an application: pressing buttons, entering text, etc.
Selendroid is used for testing Android Native as well as Hybrid applications. It tests apps using the Android driver WebView app. Tests have to be written using the Selenium 2 client API.
You do not need to modify your app while using Selendroid. You can also extend it using your own extensions during run-time. It can interact with multiple android devices, including emulators, at the same time. This feature is also owed to Selendroid’s distinct feature that you can plug and unplug devices without interrupting the test.
Using Selendroid, you can also execute parallel test cases on multiple devices. It contains an Inspector tool that can analyze the UI elements of the app that you are testing. Unlike other tools like Appium, Selendroid can find UI elements for older versions of Android, too, as early as Android 4.4 KitKat.
Selendroid features and capabilities
- Hot plugging.
- Compatibility with JSON Wire Protocol.
- Advanced User Interactions API: Support for gestures.
- Emulators are started up automatically.
- Supports different locator types.
- Supports many versions of android from Android 4.0 KitKat onwards.
- The built-in Inspector simplifies test case development.
Watch a Selendroid mobile web testing demo
6. UI Automator
UI Automator is developed and maintained by Google. It supports only Native Android apps. Tests are written in Java. Moreover, it is a black-box testing app. This essentially means that you do not need to know the internal structure of the app being tested and can rely solely on the visible UI elements.
It gives you the ability to switch between installed and system apps, enables cross-app functional testing, and lets you test multiple apps. It comes with two sets of APIs; one allows it to manipulate the app’s UI components, while the other enables it to perform various tasks on the device.
UI Automator features and capabilities
- UI Automator Viewer: GUI tool to scan current UI components on the screen.
- Performs interactions on system apps and user apps.
- Performs actions such as opening app launcher in test device.
- Simulate buttons on the screen, such as the home button.
Espresso is also a useful testing tool for the automated testing of Android UI apps. App developers and testers widely use it. This solution is also from Google. Its tests are quick to execute and reliable.
On the other hand it’s not trivial to create tests. You need a lot of tweaks to really to get them stable. We covered some of the issues of Espresso, i.e. about idling resources and thread sleep.
Espresso uses white-box testing, which is responsible for its fast speed. This means that you must be well-aware of the internal structure of the application while using Espresso to test it. Tests have to be written in Java – as with most Google-maintained frameworks.
Furthermore, it’s an open-source framework; hence you can easily extend it to better suit your needs. Another benefit of Espresso is that it allows you to write user interface tests for your application.
You can even create and record a scenario of interacting with the device. Capture snapshots to confirm the behavior of UI elements of the app. Espresso will use the recording and generate UI test cases for your app.
Espresso features and capabilities
- Tests can be built-in APK form, which can be installed and used quickly.
- Does not communicate with a server.
- Provides very stable tests thanks to the synchronized method of executing tests.
- Works flawlessly with Android Studio IDE.
Read more about Android testing with Espresso, or about the differences between Espresso vs Robotium.
MonkeyRunner is a popular tool for running automated functional tests on Android applications. The tests are written in Python.
MonkeyRunner gives you multiple device control. You can run tests across multiple devices or emulators at once, even by physically attaching them. It was developed to test applications at functional level, but you can use it to do more than just that.
This tool can also carry out something known as Regression Testing. This method of testing involves running an application and comparing the output to an already known set of correct outputs.
It also comes with an API that allows it to control functions of the device. Since MonkeyRunner uses Python, you can create modules and programs to control Android devices. Furthermore, you can extend MonkeyRunner with plugins and therefore expand its usability.
MonkeyRunner features and capabilities
- Multiple-device support.
- Functional testing.
- Tests can even be written using a recording tool.
- Supports Regression Testing.
As you see, each tool has its own advantages. To make the choice easier for you, here’s an infographic that can also help you decide on some important factors.
This image helps you to get a better picture of the current situation and choose the Android test automation framework that is the best for your project:
How to choose the best Android test automation framework
Amongst the criteria of selecting the best framework are:
- Supported platforms
- Ease of use and setup
- Ability to simulate user interactions
- Support for testing various UI elements
- Integration with development and continuous integration tools
Choosing the right Android testing tool can be a challenge. There is not a single best UI test automation tool for each case. What “the best” automation tool is depends clearly on your requirements. Here is an overview of the parameters you should consider when it comes to Android testing tools.
However, if you are about to decide on a set of testing tools to commit to, it’s worth to spend some time investigating your requirements and then look for the right tools. A test setup is usually not changed every week, you might even stick with it for the upcoming years.
So, here are the questions you can ask yourself before making your choices:
Do I need to have Continuous Integration support?
When we bring up the subject of “test automation” in front of our clients, it usually triggers discussions about Jenkins, how regularly and on which branches to automatically run the tests on and if this shouldn’t be done the same way the iOS team is doing it. And often things end with “let’s do this, that’s super important, but let’s do it later”.
Everybody is excited about fully automated workflows, but when you get into planning the details of it you might realize that it’s more work than you initially thought. This might stop you from implementing automated testing at all. Wee have been there many times with our clients: Either it gets postponed week after week or it just gets wiped off the table. And it’s clear. The whole team is busy and would rather get on with their own work instead of implementing a process which will start to pay off weeks from now.
But don’t panic: Tests are also useful when they are being run locally, without any continuous integration. And you can always work on further automating your workflow later.
Do I need to pay for a device farm to test my app?
A lot of apps deal mostly with showing text and image content. The Android Framework offers a hardware abstraction layer which works pretty well. So for many apps it does not make much sense to test on dozens of devices: The testing results will be pretty much the same on every device.
However, there are circumstances when you should strongly consider running your tests in a testing cloud on as many devices as possible:
- Your app takes advantage of camera features. Camera implementations of android are heavily dependent on the device hardware
- Your app is running native code (C++) such as MapViews (MapBox) or makes use of 3D widgets (3D content rendered natively). We have run into several issues with specific devices not showing native code properly or even crashing the app.
- Your app under test is dealing with specific sensors
If you answered yes to any of the above points, you might want to check out cloud test services such as Soucelabs, Bitbar, Experitest or Firebase testlab. All of them support running Espresso tests. Soucelabs, Bitbar and Experitest also allow you to run your Appium tests.
What if I don’t have access to the source code of the app under test?
If you are running a testing agency your clients might not trust you enough to make the source code accessible to you. Or it’s a legal issue. Or you just want to do black box testing anyway.
In any of those cases you will need a black box testing framework:
- Monkey Runner
Do I need to test native (C++, OpenGL) content in myAndroid app?
There are apps which show content in native (C++) widgets (Java native interface / JNI). Think of asserting a pin inside a MapView (Google Maps, MapBox) or checking if your 3D rendered avatar shows up in a 3D (OpenGL) rendered view. Most testing tools are not able to “look into” the OpenGL rendered view. The Android API just does not provide a way to do that.
There are still Android UI testing tools which allow you to make assertions on a pixel level:
- Monkey Runner
There is also the spoon screenshot tool which simplifies the creation and presentation of screenshots over many devices.
Do I have enough developer resources allocated to set up my testing environment?
Don’t let anyone tell you that setting up a testing environment is quick and easy. Following requirements will have a direct impact on the complexity and thus cost of your testing setup:
- Starting test runs automatically when source code is changed. You basically need a CI (Continuous Integration) setup. The quickest way to do that is to use one of the above cloud testing providers.
- You want to run the tests on your own server, you will need to look into setting up Jenkins, Teamcity or a similar software.
- You want to have a cross platform testing solution (iOS + Android) and reuse code. This will require to synchronize application development between your iOS and Android devs
- You want to test every single use case of your app. It’s gonna be hard because some things are really hard to test. The 80:20 rule will apply also in this case: With 20% of costs you will be able to cover 80% of your use cases, and vice versa.
Each of the above points will add costs for you. Some companies fail to implement testing at all because the big leap is postponed week after week and month after month.
But you can start with a simple setup. Start writing some Espresso tests today if you got developer resources or use no-coding tools such as Repeato to save on developer costs.
Do I need to test a multi-app user flow?
Sometimes you want to have a real end-to-end test, like a user registering and receiving a SMS clicking on a link, filling out a form and then returning into the app.
In this case, you should check out UIAutomator and Repeato which allow you to test use cases including app switching.
Do I want to test pixel perfect rendering?
This is a quite specific requirement, but it can make sense to check if every pixel on your screen is in the right place. Think of your team modifying styles and resources which are using in different sections of your app. It’s easy to run in a situation where a style is changed and thus causing another part of your app to look (or even behave) differently.
Checking on a pixel level will make sure that everything is rendered as it used to be in a previous release.
Here is a tutorial which shows how to setup automated pixel tests with Espresso. Also Repeato can be used to make pixel perfect assertions.
An Android test automation framework for assuring high-quality is a crucial development step that often gets not enough attention because it seems painful at first. However, if you are aware of the testing requirements and can build the testing processes accordingly, you’ll make sure to ship flawless apps.
There are numerous frameworks available to assist with Android testing; This quite elaborate post on the subject essentially boils it down to: You might need a set of tools because there is no jack of all trades testing solution.
Tags: android, mobile testing tools, test automation