How to choose the Best Android UI Testing Tool in 2020

5 May 2020 stoefln Leave a comment Uncategorized

Choosing the right Android UI Testing tool can be a challenge. There is not a single best UI test automation tool for each case.ย  What “the best” automation tool is depends clearly on your requirements.
There are a lot of parameters you should consider when it comes to UI testing.

The best Android UI testing tool? TL;DR

This is a quite elaborate post on the subject and essentially it boils down to: You might need a set of tools because there is no jack of all trades testing solution. If you don’t want to read all of it, you can skip to the diagram to get a good overview.

Here are the topics we will discuss in detail:

  1. Do I need to have Continous Integration support?
  2. Do I need to pay for a device farm to test my app?
  3. What if I don’t have access to the source code of the app under test?
  4. Do I want to test native (C++, OpenGL) content in my android app?
  5. Do I have enough developer resources allocated to set up my testing environment?
  6. Do I need to test a multi-app user flow?
  7. Do I want to test pixel perfect rendering?
  8. So, which ones are the right UI testing tools for my Android app then?
  9. How much does a UI testing cost?

 

Android UI Testing toolset

However, if you are about to decide on a set of UI testing tools to commit to, it’s worth to spend some time investigating your requirements and then look for the right tools. A test setup is usually not changed every week, you might even stick with it for the upcoming years.
So, here are the questions you can ask yourself before making your choices:

Do I need to have Continuous Integration support?

When I bring up the subject of “test automation” in front of my clients, it usually triggers discussions about Jenkins, how regularly and on which branches to automatically run the tests on and if this shouldn’t be done the same way the iOS team is doing it. And often things end with “let’s do this, that’s super important, but let’s do it later”.

Everybody is excited about fully automated workflows, but when you get into planning the details of it you might realise that it’s more work than you initially thought. This might stop you from implementing automated testing at all. I have been there many times with my clients: Either it gets postponed week after week or it just gets wiped off the table. And it’s clear. The whole team is busy and would rather get on with their own work instead of implementing a process which will start to pay off weeks from now.

How to avoid discussions about UI test automation

But don’t panic: Tests are also useful when they are being run locally, without any continuous integration. And you can always work on further automating your workflow later.

Do I need to pay for a device farm to test my app?

A lot of apps deal mostly with showing text and image content. The Android Framework offers a hardware abstraction layer which works pretty well. So for many apps it does not make much sense to test on dozens of devices: The testing results will be pretty much the same on every device.

However, there are circumstances when you should strongly consider running your tests in a testing cloud on as many devices as possible:

  1. Your app takes advantage of camera features. Camera implementations of android are heavily dependent on the device hardware
  2. Your app is running native code (C++) such as MapViews (MapBox) or makes use of 3D widgets (3D content rendered natively). We have run into several issues with specific devices not showing native code properly or even crashing the app.
  3. Your app under test is dealing with specific sensors


If you answered yes to any of the above points, you might want to check out cloud test services such as Soucelabs, Bitbar, Experitest or Firebase testlab. All of them support running Espresso tests. Soucelabs, Bitbar and Experitest also allow you to run your Appium tests.

What if I don’t have access to the source code of the app under test?

If you are running a testing agency your clients might not trust you enough to make the source code accessible to you. Or it’s a legal issue. Or you just want to do black box testing anyway.

In any of those cases you will need a black box testing framework:

  • Appium
  • UIAutomator
  • Monkey Runner

Do I want to test native (C++, OpenGL) content in my android app?

There are apps which show content in native (C++) widgets (Java native interface / JNI). Think of asserting a pin inside a MapView (Google Maps, MapBox) or checking if your 3D rendered avatar shows up in a 3D (OpenGL) rendered view. Most testing tools are not able to “look into” the OpenGL rendered view. The Android API just does not provide a way to do that.

There are still Android UI testing tools which allow you to make assertions on a pixel level:

  • Repeato
  • Monkey Runner

There is also theย spoon screenshot tool which simplifies the creation and presentation of screenshots over many devices.

Do I have enough developer resources allocated to set up my testing environment?

Don’t let anyone tell you that setting up a testing environment is quick and easy. Following requirements will have a direct impact on the complexity and thus cost of your testing setup:

  1. Starting test runs automatically when source code is changed. You basically need a CI (Continuous Integration) setup. The quickest way to do that is to use one of the above cloud testing providers.
  2. You want to run the tests on your own server, you will need to look into setting up Jenkins, Teamcity or a similar software.
  3. You want to have a cross platform testing solution (iOS + Android) and reuse code. This will require to synchronise application development between your iOS and Android devs
  4. You want to test every single use case of your app. It’s gonna be hard because some things are really hard to test. The 80:20 rule will apply also in this case: With 20% of costs you will be able to cover 80% of your use cases, and vice versa.

Each of the above points will add costs for you. Some companies fail to implement testing at all because the big leap is postponed week after week and month after month.

But you can start with a simple setup. Start writing some Espresso tests today if you got developer resources or use no-coding tools such as repeato to save on developer costs.

Do I need to test a multi-app user flow?

Sometimes you want to have a real end-to-end test, like a user registering and receiving a SMS clicking on a link, filling out a form and then returning into the app.
In this case, you should check out UIAutomator and repeato which allow you to test usecases including app switching.

Do I want to test pixel perfect rendering?

This is a quite specific requirement, but it can make sense to check if every pixel on your screen is in the right place. Think of your team modifying styles and resources which are using in different sections of your app. It’s easy to run in a situation where a style is changed and thus causing another part of your app to look (or even behave) differently.

Checking on a pixel level will make sure that everything is rendered as it used to be in a previous release.

Here is a tutorial which shows how to setup automated pixel tests with espresso.
Also repeato can be used to make pixel perfect assertions.

So, which ones are the right UI testing tools for my Android app then?

While doing research for this article, I realised that it’s really difficult to get a good overview over the tools out there. There is an obvious reason for the huge amount of android UI testing tools: They all have their specific niche.

This image might help you to get a better picture of the current situation:

Choosing the best android UI testing tool

How much does UI testing cost?

The costs will obviously depend on the tools you chose. Here is some overview which might help you to calculate your cost if you want to know the cost of a full fledged device cloud setup:

Tool

Costs for setup

Costs per month

Tech know-how needed

Complexity

Integrations

Jenkins (self hosted)

- 3 workdays for initial test creation*
- 2 workdays jenkins installation
- 5 workdays testing integration

- 1  workday server administration
- Cost for hosting


  • scripting tests 

  • setting up a server environment

  • integrating tests into CI

  • delivering test results to team (slack / email)

๐ŸŽ“๐ŸŽ“๐ŸŽ“๐ŸŽ“๐ŸŽ“ 

Insert Custom HTML

- 3 workdays for initial test creation*
- half a workday for cloud service setup

$1  per hour per virtual device

$5 per hour per real device

scripting tests (kotlin/java)

๐ŸŽ“๐ŸŽ“ ๐ŸŽ“ 

- 3 workdays for initial test creation*
- half a workday for cloud service setup

starting from โ‚ฌ199*

scripting tests (kotlin/java/c#/python/...)

๐ŸŽ“๐ŸŽ“ 

- 3 workdays for initial test creation*
- half a workday for cloud service setup

starting from โ‚ฌ159*

scripting tests (kotlin/java/c#/python/...)

๐ŸŽ“๐ŸŽ“ 

- 3 workdays for initial test creation*
- half a workday for cloud service setup

starting from โ‚ฌ298*

scripting tests (kotlin/java/c#/python/...)

๐ŸŽ“๐ŸŽ“ 

- 1 workday for inital test setup*

$1  per hour per virtual device
$5 per hour per real device

-

๐ŸŽ“ 

- 1 workday for initial test creation*

starting from $29*

-

๐ŸŽ“ 

* To make the android ui testing tools comparable I made following assumptions:

  • Price per month
  • Test automation support
  • Unlimited testing units
  • 1 User access
  • Real devices & emulators

Further I assumed an initial workload for the creation of 10 UI tests.

Tags: , ,

Like this article? thereโ€™s more where that came from.

Leave a Reply

Your email address will not be published. Required fields are marked *