Hi, it’s Stephan writing – founder of Repeato. By the end of 2018 I was once again struggling with broken Espresso tests. Frustration took over and I realized that I’ve always had a bad relationship with testing, test automation and test frameworks.
Now, 2 years later, I am dealing with the least favorite topic of software development almost full time.
What I don’t like about traditional test automation
Don’t get me wrong, I don’t want to make an “You did it all wrong” argument. The tools out there are the best we currently have, and they work very well for many of us. But I am not alone with my frustration about the status quo in test automation. In my 8 years of being an Android freelancer for many agencies and startups, I found that a “Mobile test automation doesn’t pay off” attitude is quite standard.
10 reasons why testing on Android is hard:
- Test automation setup is not trivial. It is for a very basic app. But often that’s not the case (idling resources, Dagger2, Retrofit2, RxJava2 and MockWebServer ← the length of this very well written article tells you a lot about the complexity of the setup)
- Fixing a test is hard, if you don’t fix it right away. Often I noticed a broken test too late → hassle to find the issue. So a process is required which assures regular execution of the tests
- Usually the answer to the point above is: “Continuous Integration”. If you have tried to set up CI, emulators, test automation, notifications and reports on your own server you know: It ain’t easy. It’s a huge investment
- Tests break easily and often. With shorter release cycles and constant design tweaks software products change all the time → continuous effort to maintain tests. This is especially true for early stage startups which pivot every other week
- IDs are unique right? Turn’s out they are often not → broken tests
- Even though IDs shouldn’t change, they often do → broken tests
- Looking up IDs when writing tests can be really time consuming.
Especially when you did not develop the app yourself and you don’t know the layout files of your app
- When tests break it’s often really time consuming to figure out what the problem was. Often all you get is a stacktrace
- Some things are almost impossible to test: Navigation or widgets rendered
differently (C++ content, 3D content, Webview, Animations, Maps, …)
- Rewriting a test for the 3rd time is just insanely tedious
Test automation frameworks are stupid
Bold statement right? I know that a lot of brain power has manifested in the 7200 commits of Appium. However: A lot of this development is needed to provide just a basic test foundation (such as a way to communicate with the test subject and the very specifics for the different platforms).
So what I am trying to say is: The implementation of frameworks might be complex, but essentially the principle for most testing frameworks is still like:
- Lookup an element by a single “invariant property” (such as ID)
- Interact with that element
- Lookup another element
- Check if property x of that element is like you expect it to be
And so there is still a lot of intelligent work to be taken care of by us developers. Things like:
- Mocking objects (which requires intelligently laid out software architecture)
- Synchronising processes such as background tasks or network requests
- Integrating test execution into the CI, reporting test results via APIs etc
- Fixing ID lookups etc (see above)
The way I see it, the situation of software-testing currently is:
- Test creation is complicated (need for highly skilled people) -> expensive
- Test maintenance is time consuming -> expensive
- Setting up test automation process is time consuming and complicated -> expensive
The result is that a lot of agencies (especially the ones building smaller apps) and startups (especially the ones which are still pivoting a lot), tend to prefer manual testing.
Why manual software testing is still a good choice
- A process is set up easily and there is a person in charge to enforce that process (cheap)
- The person in charge can work with basic tools such as a spreadsheet (cheap)
- The person does not even have to have IT knowledge (cheap)
- High confidence in testing results: No mocking, no flakiness, no test environment. What you see is what you get.
So is manual testing the way to go? Of course not. I’ve seen it many times: Features are added every day, it’s just a matter of time till testing becomes the bottle neck of the software development workflow.
What about Unit Testing?
Most of what I wrote above is related to Android UI testing. But a lot of it applies to unit testing as well. It’s complicated (architecture, highly trained people needed), tests break even more often and the confidence score of your testing result is lower compared to UI testing.
I am not against Unit Testing. It’s a necessity and as you probably know it’s not a replacement for UI testing.
For the sake of completeness I want to include this tweet:
Are we able to build more intelligent UI testing tools?
Back in 2018 I wondered: If a random student can test our app without even knowing a single word of kotlin, why isn’t there software which can? I mean we are about to have autonomous cars and drones right?
So I sat down and tried to imagine the cognitive work of a software tester:
- have some basic understanding of smart phone UIs (screens, transitions, lists, history back)
- have some basic understanding of the type of app s/he is testing
- have some instructions on things to check (use cases)
- know how the app should work ideally
- know how to make a screenshot and report if the app didn’t show the expected behavioor
Sounds pretty basic right? At least that’s what I thought back then. I didn’t even check for existing AI testing frameworks (if you are interested, here is a goodarticle about AI testing).
So I guess without my naivety I wouldn’t have started this project. There was one more motivating factor though: Before I wrote my first line of code, I scheduled 10 meetings with different mobile app agencies and startups here in Vienna. I asked them about their experience with test automation and if they felt the same pain. The results I got from those interviews where very convincing.
So it’s now about 18 months I have spent developing the UI testing tool and it’s called Repeato.
Repeato tries to operate much like a human: It looks at your device screen, tries to recognize visual elements, it waits for transition animations to finish before executing the next step, just as a normal user would do. And it doesn’t care about your IDs or the framework in use.
A lot of the AI part of the software is still under development, but Repeato is already used productively by happy clients 🥳