20 March 2020 Leave a comment Test automation, Tools & Frameworks
Ensuring that your app is bug-free and responsive across different devices and platforms can be challenging. This is where automated testing frameworks come in handy. In this article, we’ll explore the top Android test automation frameworks, their features and limitations, and how they compare to one another, helping you select the best framework for your project needs.
Table of Contents
Why to automate testing on Android
The importance of Android automation testing cannot be overstated. Decent testing ensures that an app’s user interface functions correctly and is responsive across different devices and platforms. With a well-functioning UI, users are more likely to have a positive experience with the app, leading to increased engagement, retention, and revenue.
Additionally, detecting and fixing bugs early in the development cycle saves time and resources in the long run.
Thus, the questions of manual vs. automated testing pops up. Automated testing frameworks optimize the testing process, making it faster and more efficient, allowing developers to focus on other aspects of the app’s development.
Ultimately, incorporating automated testing into the development process is critical to delivering a high-quality Android app that meets user expectations and drives business success.
The best Android test automation frameworks
To make choosing the best solution for Android automation testing much easier for you, we have compiled a list of the most commonly used tools; scroll down for in-depth information on each of these tools.
By the way, you might have also heard about TestProject.io which used to be pretty famous in the area – that tool has been shut down late 2022.
Repeato
Repeato is a test automation framework for Android. Tests can be recorded using a recorder and then played back. More specifically: When recording, small “fingerprints” are recorded in the vicinity of the mouse pointer. On replay, these are used to find the corresponding UI element again.
Repeato is a no-code testing tool that allows you to create tests via record & play – thus, the test author does not need coding skills. It’s ideal for automating testing processes that you would otherwise need to manually repeat. Thus, testing becomes feasible for people without coding skills; also, thanks to computer vision, tests are very robust. When needed, tests can be recreated quickly.
Repeato works based on computer vision and machine learning to recognize and interact with the elements in your user interface.
There is also the possibility of creating “Script Steps” which allow you to execute ADB or other terminal commands.
That’s how you record a test:
Afterwards, the recorded steps can then be played back:
Tests created with Repeato are stored in the file system and can also be versioned via Git as well as shared and edited within your team.
There is a free version, but advanced features such as continuous integration support or workspaces are not included.
The framework is particularly helpful as an extension for manual testing: Since developer resources are often scarce, it makes sense to outsource part of the test automation to the QA team.
Repeato features and capabilities
- Creating tests via a test recorder only takes few minutes
- Blackbox testing (no code required)
- Testing native widgets (GPU rendered things like maps or 3D content)
- Automatically check memory usage
Advantages of Repeato
- Very easy to use, can be used by non-developers (QA team)
- Repeato does not need access to the source code of the app
- Installation of the software only takes a few minutes
- Visual reporting dramatically simplifies troubleshooting
Cons of Repeato
- no open source
- extended version is commercial (49 EUR per month)
Repeato is free for small projects, have a look and simply try it out.
Appium
Appium is probably the most popular Android automation testing tool that there is. It is an open-source tool. It’s free and supports a wide range of programming languages. These include Java, Java-Script, Objective-C, Ruby, Python, and many more.
Although Appium is a cross-platform tool, we will be discussing its applicability with Android here. It uses either UIAutomater or Instrumentation for Android, and these frameworks are used under Selenium WebDriver, which gives developers the luxury of being able to use several languages.
Furthermore, Appium allows you to test many types of mobile applications. This includes Native, Hybrid, and Mobile Web. Lastly, Sauce Labs supports Appium for testing with real devices, simulators, and emulators.
Therefore, Appium is an extremely versatile platform for Android testing.
Appium features and capabilities
- No installation required.
- A vast and active community.
- The app does not need to be modified to be tested using Appium.
- Appium UI Inspector: a tool that comes with Appium to assess an app’s UI components.
Robotium
Robotium is an open-source tool for Android test automation. It was released back in 2010 and by now has become quite capable and reliable. Since then, its test execution and test readability speed have increased a lot.
Robotium makes its tests more robust by using run-time binding to UI components. It comes as a jar file that you have to compile with the app that you are testing. Not only does it support both Native and Hybrid app testing, but it also supports the basic Android UI tools such as activities, menus, and buttons as well as gestures.
Robotium uses Java as its language of choice. Using Robotium, developers can write functional tests, system tests as well as acceptance test scenarios that cover a host of Android activities. You can also create Automatic GUI (Graphical User Interface) tests.
Similar to Appium, Robotium is supported by Sauce Labs, which allows for testing on real devices.
Robotium features and capabilities
- Has device control functions such as taking screenshots and changing screen orientation.
- Can run tests on emulators and real devices.
- Supports multiple testing at the same time.
- Robotium recorder: A paid plugin to speed up the testing process.
Calabash
Calabash is another open-source testing framework for Android that works for both Native and Hybrid apps. It allows you and even people without coding skills to conduct automated acceptance tests on apps. Xamarin created it.
Calabash supports many languages, including Java, Flex, Ruby, and .NET. It has libraries that enable test scripts to interact with Native and Hybrid applications. You can configure your Calabash tests to run on many different Android devices smoothly. They provide real-time feedback and validation.
Furthermore, Calabash works for Behavior-Driven Development (BDD). This is slightly different and more advanced than Test-Driven Development. Essentially, in BDD, tasks are created in which the intended behavior of the app is specified. The idea behind BDD is that the software is built right from the vision of the owner without the developer having to interpret how it should behave.
Calabash features and capabilities
- Configuration facility to make tests compatible with a wide range of devices.
- Grants access to the Xamarin test cloud.
- Supports Cucumber (a testing tool for BDD).
- Cucumber Automation lets you write tests in simple English.
- Enables automatic UI interactions within an application: pressing buttons, entering text, etc.
Selendroid
Selendroid is used for testing Android Native as well as Hybrid applications. It tests apps using the Android driver WebView app. Tests have to be written using the Selenium 2 client API.
You do not need to modify your app while using Selendroid. You can also extend it using your own extensions during run-time. It can interact with multiple android devices, including emulators, at the same time. This feature is also owed to Selendroid’s distinct feature that you can plug and unplug devices without interrupting the test.
Using Selendroid, you can also execute parallel test cases on multiple devices. It contains an Inspector tool that can analyze the UI elements of the app that you are testing. Unlike other tools like Appium, Selendroid can find UI elements for older versions of Android, too, as early as Android 4.4 KitKat.
Selendroid features and capabilities
- Hot plugging.
- Compatibility with JSON Wire Protocol.
- Advanced User Interactions API: Support for gestures.
- Emulators are started up automatically.
- Supports different locator types.
- Supports many versions of android from Android 4.0 KitKat onwards.
- The built-in Inspector simplifies test case development.
Watch a Selendroid mobile web testing demo
UI Automator
UI Automator is developed and maintained by Google. It supports only Native Android apps. Tests are written in Java. Moreover, it is a black-box testing app. This essentially means that you do not need to know the internal structure of the app being tested and can rely solely on the visible UI elements.
It gives you the ability to switch between installed and system apps, enables cross-app functional testing, and lets you test multiple apps. It comes with two sets of APIs; one allows it to manipulate the app’s UI components, while the other enables it to perform various tasks on the device.
UI Automator features and capabilities
- UI Automator Viewer: GUI tool to scan current UI components on the screen.
- Performs interactions on system apps and user apps.
- Performs actions such as opening app launcher in test device.
- Simulate buttons on the screen, such as the home button.
Espresso
Espresso is also a useful testing tool for the automated testing of Android UI apps. App developers and testers widely use it. This solution is also from Google. Its tests are quick to execute and reliable.
On the other hand it’s not trivial to create tests. You need a lot of tweaks to really to get them stable. We covered some of the issues of Espresso, i.e. about idling resources and thread sleep.
Espresso uses white-box testing, which is responsible for its fast speed. This means that you must be well-aware of the internal structure of the application while using Espresso to test it. Tests have to be written in Java – as with most Google-maintained frameworks.
Furthermore, it’s an open-source framework; hence you can easily extend it to better suit your needs. Another benefit of Espresso is that it allows you to write user interface tests for your application.
You can even create and record a scenario of interacting with the device. Capture snapshots to confirm the behavior of UI elements of the app. Espresso will use the recording and generate UI test cases for your app.
Espresso features and capabilities
- Tests can be built-in APK form, which can be installed and used quickly.
- Does not communicate with a server.
- Provides very stable tests thanks to the synchronized method of executing tests.
- Works flawlessly with Android Studio IDE.
Read more about Android testing with Espresso, or about the differences between Espresso vs Robotium.
What’s new in the domain of test automation frameworks for Android?
Top on the list of the most used testing frameworks for Android is still Espresso, so I’ll give a brief overview of developments, but then also introduce some new test automation frameworks.
You can also check out the most common Android test automation challenges before you dive into this article.
Kaspresso
Kaspresso is a test automation framework built on top of Espresso and UIAutomator. With a lot of useful additions. It was developed to improve some of the known weaknesses of espresso (flakiness, no ADB support, difficult to read code).
The Kakau Library was used to ensure better readability.
Let’s take a look how Kakau compares to pure Espresso. Here is a simple Espresso test example:
@Test
fun logout() {
onView(withId(R.id.logoutButton))
.check(ViewAssertions.matches(
ViewMatchers.withEffectiveVisibility(ViewMatchers.Visibility.VISIBLE)))
onView(withId(R.id.logoutButton)).perform(click())
}
And here the same test in Kakau DSL syntax:
@Test
fun testFirstFeature() {
mainScreen {
logoutButton {
isVisible()
click()
}
}
}
Because longer tests can become confusing, Kaspresso’s team has extended the Kakau DSL. This way we can also don’t have to care about what is under the hood of our tests, it’s either Espresso or UI Automator.
A full test written with Kaspresso could look like this:
@Test
fun shouldPassOnNoInternetScanTest() =
beforeTest {
activityTestRule.launchActivity(null)
}.afterTest {
// cleanup
}.run {
step("Open Next Screen") {
MainScreen {
nextButton {
isVisible()
click()
}
}
}
step("Click logout button to log out") {
SimpleScreen {
logoutButton {
isVisible()
click()
}
}
}
...
The speed comparison with a conventional UIAutomator (Appium) Test is also remarkable:
Kaspresso chooses a very sophisticated strategy (compared to Espresso or Barista) to improve the stability (flakiness) of tests: Android’s event queue is monitored and commands are only sent when Android is in a calm state and there is a low likeliness that UI commands will be skipped because of lack of computation resources. In addition, the testing framework also automatically closes system dialogs that sometimes cause a test to fail because they cover important elements of the app.
If you want to know more, the project and functions are really well documented.
Advantages of Kaspresso
- Greatly improved stability for tests compared to all other test automation frameworks
- Improved speed
- Abstraction and thus better maintainability for tests
- Open Source (110 PullRequests & 887 stars on Github)
Cons of Kaspresso
- DSL adds a further level of abstraction and requires some time to get to know and familiarize yourself with
- DSL can sometimes lead to confusing code. See an example here.
- Documentation and some issues are partly in Russian
Barista
Barista is a big extension of espresso. The stability (flakiness) is improved with a retry strategy. For certain tests, you can specify how often a test should be repeated if it fails. In addition, like Kaspresso, Barista uses an abstraction applied to PageObjects for UI components and test routines.
A simple example:
@Test
fun shouldShowRecommendedOffers_whenComingBackAfterLogin() {
HomeListScreenObject()
.clickSearchToolbar()
.search("Java")
.clickFloatingLoginButton()
.fillCredentialsAndLogIn()
.assertThat { headerContains("10 ofertas recomendadas") }
}Fairly legible right? Obviously, some more work is needed to implement those methods. But still, the structure is very clean and readable:
class HomeListScreenObject : NavigationScreenObject() {
fun clickSearchToolbar(): SearchScreenObject {
clickOn(R.id.toolbar)
return SearchScreenObject()
}
fun assertThat(assertionBlock: HomeListScreenAssertions.() -> Unit): HomeListScreenObject {
HomeListScreenAssertions().assertionBlock()
return this
}
}
class HomeListScreenAssertions : BaseScreenAssertions(R.id.homeListRoot) {
fun headerContains(text: String) {
assertDisplayed(text)
}
}
class SearchResultScreenObject : BaseScreenObject {
fun clickFloatingLoginButton(): LoginScreenObject {
clickOn(R.id.searchResultLoginButton)
return LoginScreenObject()
}
fun clickFilters(): SearchFiltersScreenObject {
clickOn(R.id.searchResultFilterButton)
return SearchFiltersScreenObject()
}
fun assertThat(assertionBlock: SearchResultScreenAssertions.() -> Unit): SearchResultScreenObject {
SearchResultScreenAssertions().assertionBlock()
return this
}
}
class SearchResultScreenAssertions : BaseScreenAssertions(R.id.searchResultRoot) {
fun createAlertButtonIsDisplayed() {
assertDisplayed(R.id.searchResultCreateAlertButton)
}
fun searchBarContains(text: String) {
assertAnyView(
viewMatcher = withParent(withId(R.id.toolbar)),
condition = withText(text)
)
}
}
In case of a feature change in the app you might be able to avoid changing all your 50 tests and instead only change the implementation of a single test method.
Advantages of Barista
- Improved stability through auto retry and auto scroll
- Good readability thanks to an additional level of abstraction (Page objects)
- Very good documentation, easy to use
- Open source and active community (220 pull requests & 1222 stars on Github)
Disadvantages of Barista
- Idling resources still have to be implemented manually
BusyBee
BusyBee is also an extension of Espresso. More specifically, a substitute for CountingIdlingResource is used, which has some known difficulties. One of them being debugging: If a test fails because CountingIdlingResource does not signal an idling state, it is often not clear why the exception happened.
Instead of just tracking the number of ongoing processes, BusyBee logs the specific, currently running processes. This is accomplished by tracking a Java object, a character string or an ID.
If a timeout occurs, the list of running processes can be output. This way debugging is getting easier.
Advantages of BusyBee
- Simple framework that simplifies the handling of tiresome IdlingResources
- Open Source (11 pull requests, 143 stars)
Cons of BusyBee
- Small solution, no substitute for barista or kaspresso
Waldo
Waldo is a new cloud service that allows you to record tests using a recorder. The developers have focused on offering a simple interface that also allows non-developers to create tests. The whole thing works, just as with Repeato, without a line of code.
A visual comparison of the original test run (everything worked) with the faulty test run (exception happened) makes debugging a breeze:
Disclaimer: I encountered some difficulties while testing the service:
- The instructions on how to export the build were not visible (empty dialog)
- When starting the build, an error occurred several times – unfortunately that’s when I ran out of patience. TBC…!
Advantages
- Very clear UI, easy to use
- Great animation, the llama signup is really fun!
Cons
- Price ($ 1999 USD / month)
- more complex use cases or data driven testing not possible
- Currently no Android support (“Coming soon”)
- The service does not seem to be over beta stage yet
MonkeyRunner
MonkeyRunner is a popular tool for running automated functional tests on Android applications. The tests are written in Python.
MonkeyRunner gives you multiple device control. You can run tests across multiple devices or emulators at once, even by physically attaching them. It was developed to test applications at functional level, but you can use it to do more than just that.
This tool can also carry out something known as Regression Testing. This method of testing involves running an application and comparing the output to an already known set of correct outputs.
It also comes with an API that allows it to control functions of the device. Since MonkeyRunner uses Python, you can create modules and programs to control Android devices. Furthermore, you can extend MonkeyRunner with plugins and therefore expand its usability.
MonkeyRunner features and capabilities
- Multiple-device support.
- Functional testing.
- Tests can even be written using a recording tool.
- Supports Regression Testing.
As you see, each tool has its own advantages. To make the choice easier for you, here’s a guideline to help you decide on some important factors.
How to choose the best Android test automation framework
Amongst the criteria of selecting the best framework are:
- Supported platforms
- Ease of use and setup
- Ability to simulate user interactions
- Support for testing various UI elements
- Integration with development and continuous integration tools
Choosing the right Android testing tool can be a challenge. There is not a single best UI test automation tool for each case. What “the best” automation tool is depends clearly on your requirements. Here is an overview of the parameters you should consider when it comes to Android testing tools.
However, if you are about to decide on a set of testing tools to commit to, it’s worth to spend some time investigating your requirements and then look for the right tools. A test setup is usually not changed every week, you might even stick with it for the upcoming years.
So, here are the questions you can ask yourself before making your choices:
Do I need to have Continuous Integration support?
When we bring up the subject of “test automation” in front of our clients, it usually triggers discussions about Jenkins, how regularly and on which branches to automatically run the tests on and if this shouldn’t be done the same way the iOS team is doing it. And often things end with “let’s do this, that’s super important, but let’s do it later”.
Everybody is excited about fully automated workflows, but when you get into planning the details of it you might realize that it’s more work than you initially thought. This might stop you from implementing automated testing at all. Wee have been there many times with our clients: Either it gets postponed week after week or it just gets wiped off the table. And it’s clear. The whole team is busy and would rather get on with their own work instead of implementing a process which will start to pay off weeks from now.
But don’t panic: Tests are also useful when they are being run locally, without any continuous integration. And you can always work on further automating your workflow later.
Do I need to pay for a device farm to test my app?
A lot of apps deal mostly with showing text and image content. The Android Framework offers a hardware abstraction layer which works pretty well. So for many apps it does not make much sense to test on dozens of devices: The testing results will be pretty much the same on every device.
However, there are circumstances when you should strongly consider running your tests in a testing cloud on as many devices as possible:
- Your app takes advantage of camera features. Camera implementations of android are heavily dependent on the device hardware
- Your app is running native code (C++) such as MapViews (MapBox) or makes use of 3D widgets (3D content rendered natively). We have run into several issues with specific devices not showing native code properly or even crashing the app.
- Your app under test is dealing with specific sensors
If you answered yes to any of the above points, you might want to check out cloud test services such as Soucelabs, Bitbar, Experitest or Firebase testlab. All of them support running Espresso tests. Soucelabs, Bitbar and Experitest also allow you to run your Appium tests.
What if I don’t have access to the source code of the app under test?
If you are running a testing agency your clients might not trust you enough to make the source code accessible to you. Or it’s a legal issue. Or you just want to do black box testing anyway.
In any of those cases you will need a black box testing framework:
- Appium
- UIAutomator
- Monkey Runner
Do I need to test native (C++, OpenGL) content in myAndroid app?
There are apps which show content in native (C++) widgets (Java native interface / JNI). Think of asserting a pin inside a MapView (Google Maps, MapBox) or checking if your 3D rendered avatar shows up in a 3D (OpenGL) rendered view. Most testing tools are not able to “look into” the OpenGL rendered view. The Android API just does not provide a way to do that.
There are still Android UI testing tools which allow you to make assertions on a pixel level:
- Repeato
- Monkey Runner
There is also the spoon screenshot tool which simplifies the creation and presentation of screenshots over many devices.
Do I have enough developer resources allocated to set up my testing environment?
Don’t let anyone tell you that setting up a testing environment is quick and easy. Following requirements will have a direct impact on the complexity and thus cost of your testing setup:
- Starting test runs automatically when source code is changed. You basically need a CI (Continuous Integration) setup. The quickest way to do that is to use one of the above cloud testing providers.
- You want to run the tests on your own server, you will need to look into setting up Jenkins, Teamcity or a similar software.
- You want to have a cross platform testing solution (iOS + Android) and reuse code. This will require to synchronize application development between your iOS and Android devs
- You want to test every single use case of your app. It’s gonna be hard because some things are really hard to test. The 80:20 rule will apply also in this case: With 20% of costs you will be able to cover 80% of your use cases, and vice versa.
Each of the above points will add costs for you. Some companies fail to implement testing at all because the big leap is postponed week after week and month after month.
But you can start with a simple setup. Start writing some Espresso tests today if you got developer resources or use no-coding tools such as Repeato to save on developer costs.
Do I need to test a multi-app user flow?
Sometimes you want to have a real end-to-end test, like a user registering and receiving a SMS clicking on a link, filling out a form and then returning into the app.
In this case, you should check out UIAutomator and Repeato which allow you to test use cases including app switching.
Do I want to test pixel perfect rendering?
This is a quite specific requirement, but it can make sense to check if every pixel on your screen is in the right place. Think of your team modifying styles and resources which are using in different sections of your app. It’s easy to run in a situation where a style is changed and thus causing another part of your app to look (or even behave) differently.
Checking on a pixel level will make sure that everything is rendered as it used to be in a previous release.
Here is a tutorial which shows how to setup automated pixel tests with Espresso. Also Repeato can be used to make pixel perfect assertions.
Android UI testing infographic
This image helps you to get a better picture of the current situation and choose the Android test automation framework that is the best for your project:
Conclusion
An Android test automation framework for assuring high-quality is a crucial development step that often gets not enough attention because it seems painful at first. However, if you are aware of the testing requirements and can build the testing processes accordingly, you’ll make sure to ship flawless apps.
There are numerous frameworks available to assist with Android testing; This quite elaborate post on the subject essentially boils it down to: You might need a set of tools because there is no jack of all trades testing solution.
Even with Espresso continuously being improved, there is still enough potential to make test automation more practicable, maintainable and intuitiv. Therefore there are still new but also older alternative test automation frameworks that address these weaknesses. Unfortunately, there is no clear test winner. Each of the frameworks clearly has its strengths and weaknesses.
Tags: android, mobile testing tools, test automation