There are still a lot of unsolved challenges when implementing mobile test automation. This article takes a closer look at the reasons and mentions helpful strategies and tools.
Overview: the 3 main test automation pain points
We don’t make a claim that this list is complete. Please let us know what you find particularly challenging related to mobile test automation.
What is test flakiness?
Flaky tests are introduced when a testing framework is not fully aware of the internal state of the app or the operating system. Which is usually the case. Think of the following use case:
- User presses a button
- App fetches data from the server
- App shows dialog
Now, if we want to automate this, the mobile testing framework needs to wait for the dialog to show up, and then interact with the dialog afterward. Unfortunately, many testing frameworks are not aware of the background process which fetches the data and try to interact with the dialog right away. That can be alright if the server responds quickly enough. But what happens if the network connection is temporarily bad? Then the test fails, even though it was successful the 3 other times you ran your test.
That experience can be quite frustrating because often it’s hard to tell why a test failed exactly. For example, Android Espresso only delivers an error like this:
failed: android.support.test.espresso.NoMatchingViewException: No views in hierarchy found matching: with id: com.mycom.myapp:id/button_submit
What to do about flaky tests
Depending on the framework in use there are different strategies.
Android Espresso offers a mechanism called Idling Resources, which exposes parts of the internal state of the application (“calculating”, “fetching data from the server”, …) to the testing framework. However, this comes with some trade-offs.
Recently there have been some frameworks popping up which deal with this problem:
- Azimolabs Condition Watcher: It synchronizes background operations of your app with the test thread. ConditionWatcher can be used as a replacement to Espresso’s IdlingResources or it can work in parallel with them.
- Kaspresso: Framework built on top of Espresso. By monitoring Android’s event queue and sending command only when Android is ready to receive commands it reduces the likeliness that UI commands will be skipped because of lack of computation resources.
- Repeato: This tool relies on computer vision to figure out if the app is ready for the next interaction. It observes the screen and even waits till elements on the screen stopped animating before triggering the next commands.
- Barista is also based on Espresso. The stability of tests is improved by a retry strategy: You can specify for tests how often a test should be retried in case it fails.
Lack of developer resources
If you have worked in a software team, we are sure you have been in a similar place: Developer resources are always scarce. The team is busy implementing new features while trying to fix the bugs which pop up more frequently than expected. The vision to implement a full-fledged CI with mobile test automation is there, but there is always something more urgent, so plans to automate testing get postponed.
Strategies for saving developer time and resources on test automation
Besides the very obvious option of getting a QA person who takes care of testing manually, there are initiatives that you can take to save time and budget. But let’s take a closer look at the cost factors of mobile test automation:
- Cost of the setup:
- Test automation technology and architecture
- Training staff
- Creating or refactoring your code to be testable
- The initial creation of tests
- Cost of maintenance of the test automation system:
- Finding errors in broken tests
- Fixing those errors
- Compatibility problems introduced by updates or changes in one of your test automation components (e.g. update of Jenkins might require an update of your build script and an update of the testing library, etc.)
How to reduce the cost for a mobile test automation setup
1. Write testable code
This is mostly relevant for unit testing. If you don’t take care of sticking to an architecture that makes unit testing possible (mocking objects, …), then it will hold you back from unit testing later. The refactoring cost for making an outgrown product unit-testable is quite high and will introduce even more bugs initially.
2. Create a strategic and realistic plan
You probably have heard of the pareto principle and it’s also true for testing: You will be able to cover 80 percent of your app’s functionality with 20 percent of the time. To cover the rest, might be a huge effort or even impossible.
Be realistic about how many resources you have for test automation. Then come up with a list of use cases: What are the main flows in your app which need to be tested, because they are crucial to your app.
In a second step go through all your use cases and ask yourself: Which ones can be easily automated and which ones are really difficult and should rather be taken care of via manual testing? Make a simple text-based list before you start with coding!
3. Keep it simple
We all know how great it would be if everything, including testing, would be fully automated. However, we have worked for many startups and software agencies and our experience is that a fully automated testing solution that runs without maintenance is an illusion. And because many developers know that it’s a big step to set it up, it often keeps them to start automating tests in the first place. Try to keep things simple: Find a schedule on how tests can be run before every release. Even locally on a development machine. As soon as there is a reasonable amount of tests you can still work on integrating them into your CI.
4. Reduce the complexity of the software setup
Use tools that can be used by testers rather than developers: no-code test automation is getting a real thing. Let your QA team own their core topic: Testing.
5. Use of SaaS testing solutions
Setting up Jenkins and integrating automated testing on your server can be a huge step: Keep in mind that in order to run UI tests you will need an emulator or even real devices connected to your server. There are really good services which help you to set up things quickly. Of cause, it’s great to use Open Source and an in-house CI pipeline, but the cost of setting it up should not be underestimated.
How to reduce the cost for mobile test automation maintenance
1. Simplify debugging
When a test fails, usually all you get is just a stacktrace. Try to create some additional debug information by creating screenshots and log files. Some tools out there:
- Spoon allows you to create an overview of screenshots, for visual inspection of test executions across different devices.
- Facebook screenshot testing: Allows you to inspect regressions visually by comparing pixels.
- Taking a screenshot right before a test fails with Espresso
2. Run tests locally
Before pushing new code it’s a good idea to run tests right away. If a test fails, it’s quick to figure out what went wrong. If a continuous integration job runs the tests 2 days later on the server and you need to take care of checking out the right version it will be more time-consuming.
3. Run tests often
The earlier you find a broken test, the easier it will be to fix it.
4. Keep it simple
Distributed systems are much more fragile than a single piece of software. Continuous Integration is highly distributed with components that communicate with one another. Try to keep an eye on the number of components and only go for a full-fledged CI setup if you got a dedicated tech person who can take care of maintenance.
5. Make use of abstraction
There are testing concepts that allow you to increase reusability and maintainability by introducing an additional level of abstraction:
- Robot Pattern: Allows you to so separate the “what should be tested” from the “how should be tested”. You implement a robot for each screen or section in your app. The implementation contains all the interactions with your UI via the test framework API. That’s the how. Then you can use those robots to write your tests without caring about the underlying implementation. Let’s say the testing API changes OR a certain part of your app changes, you will only have to change the implementation of the robot. The tests can most often stay the same.
- Page Object Pattern: Similar to the Robot Pattern, it introduces an intermediary, higher-level API for interacting with your app. Every screen of your app is abstracted by a class which then can be easily used in your test cases. Here is a very nice tutorial on how to use The Page Object Pattern with Espresso. Kakao is an extension of Espresso which also makes use of this concept.
- Behavior Driven Development (BDD): Abstracts from the actual test instrumentation commands via an additional layer of natural language. Tests are written in natural language sticking to a simple format. Basic instructions are usually supported by the BDD framework, more complex commands need to be implemented by the developer. For Android there is a library that allows you to use the Cucumber BDD Framework.
Keeping pace with agile development
Continuous delivery is great: It allows you to build, test, and release software with greater speed and frequency. However, it can put a lot of pressure on a QA team if testing is not fully automated.
To test in an agile way requires us to make a shift in the way we test. The traditional testing pyramid looked like this:
But things have changed since then
With experimental features introduced by modern product development techniques like user center design, we are facing constantly changing requirements. AB-testing makes big parts of apps redundant. Experimental features are introduced and changed after the release. Things like that make maintenance of Unit tests particularly difficult and time-consuming.
On the other hand, smarter tools have emerged which make testing higher up the pyramid more affordable and robust. That’s why people like Kent C. Dodds propose to replace the pyramid with a “testing trophy”:
“The Testing Trophy” 🏆
— Kent C. Dodds (@kentcdodds) February 6, 2018
Strategies to cope with agile development
- Move up the testing pyramid, embrace UI testing
- Separate the why from the how of testing: Abstraction Patterns (see above) can make a big difference
- Use tools that allow you to create UI tests efficiently: Tools like Repeato or Leapwork allow you to forward UI test automation to your QA team
Even though we still see the same old challenges in test automation, tools and methodologies are constantly evolving. Test automation software is becoming smarter, which makes lives for developers and testers easier. And not just that. In the end, it has a positive impact on the experience of millions of software users.
At Repeato we try to build a test automation tool that can be used by non-technical testers, to drastically bring down the cost of test automation. There is a free version available, and we would be happy to hear about your experience using it!
Attributions: Cartoon vector created by vectorjuice – www.freepik.com