5 New Test Automation Frameworks for Android (2020)

new test automation frameworks 2020

12 November 2020 stoefln Leave a comment Uncategorized

The year is slowly coming to an end, so just the right time to look into what’s new around test automation frameworks.

Top on the list of the most used testing tools for Android is still Espresso, so I’ll give a brief overview of developments, but then also introduce some new test automation frameworks.

Here is the section overview:

What’s new in Android Espresso?

Since Android Espresso is already very well known, I will not go into the basic description of the framework here. Instead, I have compiled a list of the features that have been released in the last few months (Jan 2020 – Nov 2020):

  • Add Espresso ViewMatcher APIs for negative conditions
  • Allow unregistering Loopers from Espresso IdlingRegistry
  • Support specifying junit RunListeners via java.util.ServiceLoader
  • Support filtering by prerelease SDKs via SdkSuppress#codeName
  • Add truth extensions for Location
  • Add truth assertions for Bundle string and parceable arrays.
  • Support ActivityOptions in ActivityScenario
  • Make activity lifecycle transition timeout configurable
  • Add a BundleMatchers.isEmpty() and isEmptyOrNull() methods
  • Allow Intents.release without Intents.init
  • Add ViewMatchers.isFocused()
  • Enhance error messaging for WithIdMatcher
  • ViewMatchers.isDisplayingAtLeast() works for views with negative scale.
  • Remove spurious wait in waitForAtLeastOneActivityToBeResumed.
  • Set correct meta state for ACTION_UP

 

Image credits by developer.android.com

On top of that there is a great cheat sheet available for download here.

Nothing has changed about the tiresome topic of idling resources. These are still necessary to keep espresso up to date about whether it needs to wait before executing the next step. For example because a background process has not yet been completed.

NestedScrollViews are still not supported either, so an extra implementation is necessary.

 

 

 

Kaspresso

Kaspresso is a test automation framework built on top of Espresso and UIAutomator. With a lot of useful additions. It was developed to improve some of the known weaknesses of espresso (flakiness, no ADB support, difficult to read code).

The Kakau Library was used to ensure better readability.

Let’s take a look how Kakau compares to pure Espresso. Here is a simple Espresso test example:

@Test
fun logout() {
    onView(withId(R.id.logoutButton))
        .check(ViewAssertions.matches(
               ViewMatchers.withEffectiveVisibility(ViewMatchers.Visibility.VISIBLE)))
    onView(withId(R.id.logoutButton)).perform(click())
}

And here the same test in Kakau DSL syntax:

@Test
fun testFirstFeature() {
    mainScreen {
        logoutButton {
            isVisible()
            click()
        }
    }
}

Because longer tests can become confusing, Kaspresso’s team has extended the Kakau DSL. This way we can also don’t have to care about what is under the hood of our tests, it’s either Espresso or UI Automator.
A full test written with Kaspresso could look like this:

@Test
fun shouldPassOnNoInternetScanTest() =
    beforeTest {
        activityTestRule.launchActivity(null)
    }.afterTest {
        // cleanup
    }.run {
        step("Open Next Screen") {
            MainScreen {
                nextButton {
                    isVisible()
                    click()
                }
            }
        }

        step("Click logout button to log out") {
            SimpleScreen {
                logoutButton {
                    isVisible()
                    click()
                }
            }
        }
...

The speed comparison with a conventional UIAutomator (Appium) Test is also remarkable:

Kaspresso chooses a very sophisticated strategy (compared to Espresso or Barista) to improve the stability (flakiness) of tests: Android’s event queue is monitored and commands are only sent when Android is in a calm state and there is a low likeliness that UI commands will be skipped because of lack of computation resources. In addition, the testing framework also automatically closes system dialogs that sometimes cause a test to fail because they cover important elements of the app.

If you want to know more, the project and functions are really well documented.

Advantages of Kaspresso:

  • Greatly improved stability for tests compared to all other test automation frameworks
  • Improved speed
  • Abstraction and thus better maintainability for tests
  • Open Source (110 PullRequests & 887 stars on Github)

Cons of Kaspresso:

  • DSL adds a further level of abstraction and requires some time to get to know and familiarize yourself with
  • DSL can sometimes lead to confusing code. See an example here.
  • Documentation and some issues are partly in Russian

Barista

Barista is a big extension of espresso. The stability (flakiness) is improved with a retry strategy. For certain tests, you can specify how often a test should be repeated if it fails. In addition, like Kaspresso, Barista uses an abstraction applied to PageObjects for UI components and test routines.

A simple example:

@Test
fun shouldShowRecommendedOffers_whenComingBackAfterLogin() {
  HomeListScreenObject()
    .clickSearchToolbar()
    .search("Java")
    .clickFloatingLoginButton()
    .fillCredentialsAndLogIn()
    .assertThat { headerContains("10 ofertas recomendadas") }
}

Fairly legible right? Obviously, some more work is needed to implement those methods. But still, the structure is very clean and readable:

class HomeListScreenObject : NavigationScreenObject() {

  fun clickSearchToolbar(): SearchScreenObject {
    clickOn(R.id.toolbar)
    return SearchScreenObject()
  }

  fun assertThat(assertionBlock: HomeListScreenAssertions.() -> Unit): HomeListScreenObject {
    HomeListScreenAssertions().assertionBlock()
    return this
  }
}

class HomeListScreenAssertions : BaseScreenAssertions(R.id.homeListRoot) {
  fun headerContains(text: String) {
    assertDisplayed(text)
  }
}

class SearchResultScreenObject : BaseScreenObject {

  fun clickFloatingLoginButton(): LoginScreenObject {
    clickOn(R.id.searchResultLoginButton)
    return LoginScreenObject()
  }

  fun clickFilters(): SearchFiltersScreenObject {
    clickOn(R.id.searchResultFilterButton)
    return SearchFiltersScreenObject()
  }

  fun assertThat(assertionBlock: SearchResultScreenAssertions.() -> Unit): SearchResultScreenObject {
    SearchResultScreenAssertions().assertionBlock()
    return this
  }
}

class SearchResultScreenAssertions : BaseScreenAssertions(R.id.searchResultRoot) {

  fun createAlertButtonIsDisplayed() {
    assertDisplayed(R.id.searchResultCreateAlertButton)
  }

  fun searchBarContains(text: String) {
    assertAnyView(
      viewMatcher = withParent(withId(R.id.toolbar)),
      condition = withText(text)
    )
  }
}

In case of a feature change in the app you might be able to avoid changing all your 50 tests and instead only change the implementation of a single test method.

Advantages of Barista:

  • Improved stability through auto retry and auto scroll
  • Good readability thanks to an additional level of abstraction (Page objects)
  • Very good documentation, easy to use
  • Open source and active community (220 pull requests & 1222 stars on Github)

Disadvantages of Barista:

  • Idling resources still have to be implemented manually

 

BusyBee

BusyBee is also an extension of Espresso. More specifically, a substitute for CountingIdlingResource is used, which has some known difficulties. One of them being debugging: If a test fails because CountingIdlingResource does not signal an idling state, it is often not clear why the exception happened.

Instead of just tracking the number of ongoing processes, BusyBee logs the specific, currently running processes. This is accomplished by tracking a Java object, a character string or an ID.
If a timeout occurs, the list of running processes can be output. This way debugging is getting easier.

Advantages of BusyBee:

  • Simple tool that simplifies the handling of tiresome IdlingResources
  • Open Source (11 pull requests, 143 stars)

Cons of BusyBee:

  • Small tool, no substitute for barista or kaspresso

 

Repeato

Repeato test automation logoRepeato is a new NoCode test automation framework for Android. Tests can be recorded using a recorder and then played back. More specifically: When recording, small “fingerprints” are recorded in the vicinity of the mouse pointer. On replay, these are used to find the corresponding UI element again.

There is also the possibility of creating “Script Steps” which allow you to execute ADB or other terminal commands.

That’s how you record a test:

 

Afterwards, the recorded steps can then be played back:

 

Tests created with Repeato are stored in the file system and can also be versioned via Git as well as shared and edited within your team.

There is a free version, but advanced features such as continuous integration support or workspaces are not included.

The tool is particularly helpful as an extension for manual testing: Since developer resources are often scarce, it makes sense to outsource part of the test automation to the QA team.

Advantages of Repeato:

  • Very easy to use, can be used by non-developers (QA team)
  • Repeato does not need access to the source code of the app
  • Installation of the software only takes a few minutes
  • Visual reporting dramatically simplifies troubleshooting

Cons of Repeato:

  • no open source
  • extended version is commercial (49 EUR per month)

Waldo

Waldo is a new cloud service that allows you to record tests using a recorder. The developers have focused on offering a simple interface that also allows non-developers to create tests. The whole thing works, just as with Repeato, without a line of code.

Record and replay tests directly in your browser. Image credits: waldo.io

A visual comparison of the original test run (everything worked) with the faulty test run (exception happened) makes debugging a breeze:

Clean visualisation of test results. Image Credits: waldo.io

Disclaimer: I encountered some difficulties while testing the service:

  1. The instructions on how to export the build were not visible (empty dialog)
  2. When starting the build, an error occurred several times – unfortunately that’s when I ran out of patience. TBC…!Waldo fails

Advantages:

  • Very clear UI, easy to use
  • Great animation, the llama signup is really fun!

Cons:

  • Price ($ 1999 USD / month)
  • more complex use cases or data driven testing not possible
  • Currently no Android support (“Coming soon”)
  • The service does not seem to be over beta stage yet

Conclusio

Even with Espresso continuously being improved, there is still enough potential to make test automation more practicable, maintainable and intuitiv. Therefore there are still new but also older alternative test automation frameworks that address these weaknesses. Unfortunately, there is no clear test winner. Each of the frameworks clearly has its strengths and weaknesses.

Computer Vectors by Vecteezy

Tags: , , , , ,

Like this article? there’s more where that came from.