16 September 2020 Stephan Petzl Leave a comment Test automation

Hi, it’s Stephan writing – founder of Repeato. By the end of 2018 I was once again struggling with broken Espresso tests. Frustration took over and I realized that I’ve always had a bad relationship with testing, test automation and test frameworks.

Now, 2 years later, I am dealing with the least favorite topic of software development almost full time.

Why?

Because I decided to write my own test automation tool Repeato.

What I don’t like about traditional test automation

Don’t get me wrong, I don’t want to make an “You did it all wrong” argument. The tools out there are the best we currently have, and they work very well for many of us. But I am not alone with my frustration about the status quo in test automation. In my 8 years of being an Android freelancer for many agencies and startups, I found that a “Mobile test automation doesn’t pay off” attitude is quite standard.

10 reasons why testing on Android is hard:

1. Test automation setup is not trivial. It is for a very basic app. But often that’s not the case (idling resources, Dagger2, Retrofit2, RxJava2 and MockWebServer ← the length of this very well written article tells you a lot about the complexity of the setup)
2. Fixing a test is hard, if you don’t fix it right away. Often I noticed a broken test too late → hassle to find the issue. So a process is required which assures regular execution of the tests
3. Usually the answer to the point above is: “Continuous Integration”. If you have tried to set up CI, emulators, test automation, notifications and reports on your own server you know: It ain’t easy. It’s a huge investment
4. Tests break easily and often. With shorter release cycles and constant design tweaks software products change all the time → continuous effort to maintain tests. This is especially true for early stage startups which pivot every other week
5. IDs are unique right? Turn’s out they are often not → broken tests
6. Even though IDs shouldn’t change, they often do → broken tests
7. Looking up IDs when writing tests can be really time consuming.
   Especially when you did not develop the app yourself and you don’t know the layout files of your app
8. When tests break it’s often really time consuming to figure out what the problem was. Often all you get is a stacktrace
9. Some things are almost impossible to test: Navigation or widgets rendered
   differently (C++ content, 3D content, Webview, Animations, Maps, …)
10. Rewriting a test for the 3rd time is just insanely tedious

Test automation frameworks are stupid

Bold statement right? I know that a lot of brain power has manifested in the 7200 commits of Appium. However: A lot of this development is needed to provide just a basic test foundation (such as a way to communicate with the test subject and the very specifics for the different platforms).
So what I am trying to say is: The implementation of frameworks might be complex, but essentially the principle for most testing frameworks is still like:

1. Lookup an element by a single “invariant property” (such as ID)
2. Interact with that element
3. Lookup another element
4. Check if property x of that element is like you expect it to be

And so there is still a lot of intelligent work to be taken care of by us developers. Things like:

• Mocking objects (which requires intelligently laid out software architecture)
• Synchronising processes such as background tasks or network requests
• Integrating test execution into the CI, reporting test results via APIs etc
• Fixing ID lookups etc (see above)

Status Quo

The way I see it, the situation of software-testing currently is:

1. Test creation is complicated (need for highly skilled people) -> expensive
2. Test maintenance is time consuming -> expensive
3. Setting up test automation process is time consuming and complicated -> expensive

The result is that a lot of agencies (especially the ones building smaller apps) and startups (especially the ones which are still pivoting a lot), tend to prefer manual testing.

Why manual software testing is still a good choice

1. A process is set up easily and there is a person in charge to enforce that process (cheap)
2. The person in charge can work with basic tools such as a spreadsheet (cheap)
3. The person does not even have to have IT knowledge (cheap)
4. High confidence in testing results: No mocking, no flakiness, no test environment. What you see is what you get.

So is manual testing the way to go? Of course not. I’ve seen it many times: Features are added every day, it’s just a matter of time till testing becomes the bottle neck of the software development workflow.

What about Unit Testing?

Most of what I wrote above is related to Android UI testing. But a lot of it applies to unit testing as well. It’s complicated (architecture, highly trained people needed), tests break even more often and the confidence score of your testing result is lower compared to UI testing.
I am not against Unit Testing. It’s a necessity and as you probably know it’s not a replacement for UI testing.
For the sake of completeness I want to include this tweet:

Like this article? there’s more where that came from.

• Vertically Align Text to Top within a UILabel
• Understanding the ‘No bundle URL present’ Error in React Native
• Configuring Windows Service for Optimal Screen Resolution in Automated UI Testing
• Xcode 7: Bridging the Gap Between App Tests and UI Tests
• Ensuring a Fresh State for Each UI Test in Robotium