Running tests as part of your Continuous Integration

Currently, we offer two products:

• Repeato Studio: A desktop app that allows you to easily create and edit tests and test batches ➡ Documentation
• Repeato CLI: A headless Node.js test runner application, that will enable you to run test batches on a server. ➡ Documentation

We currently don’t offer any cloud services for running your tests.
However, you can use our test runner to run tests on third-party services such as Github Actions, Bitrise, Codemagic or CircleCI.

Since those services are only supporting virtual devices, we recommend also recording your tests on the same virtual devices. This way you will get the best test stability and performance.

Even though Repeato CLI will run on pretty much any server, we currently offer 3 setup guides (more to come):

• Bitrise
• Github Actions
• Your own server

Setup

Add a “Repeato Test Runner” step to your workflow in the Bitrise Workflow Editor. You can also run the “Repeato Test Runner” step directly with Bitrise CLI.

The step takes care of checking the system requirements as well as installing and starting the headless Repeato CLI test runner. After running the tests, it sets the environment variables that can be used in consecutive steps.

Configuring the Step

1. Add the Repeato Test Runner Step to your Workflow as one of the Steps in your Workflow.
2. Make sure your test workspace is checked out.
3. Set the Workspace Path to point to the repeato workspace directory.
4. Set the Batch Id. The batch ID of the test batch you want to execute.
5. Set the License Key.

Please make sure AVD or IOS emulator is configured properly. Further, your Android/iOS app build must be installed on the emulator before running the test batch. We recommend to run the tests on the same device / emulator you originally used for creating them. This way you will get the most stability and performance.

For iOS Simulator there isn’t any step, that starts the simulator, as the xcodebuild test command for iOS Testing step boots the simulator by default.

Inputs

Outputs

Tips & Troubleshooting

The emulator needs some time to boot up. The earlier you place the step in your workflow, the more tasks, such as cloning or caching, you can complete in your workflow before the emulator starts working. We recommend that you also add Wait for Android emulator Step (in the case of android) to your workflow as it acts as a shield preventing the AVD manager to kick in too early. Make sure you add the Wait for Android emulator Step BEFORE the Repeato Test Runner Step so our step can use AVD Manager.

Demo Pipeline (Android / Flutter)

Here is the our demo pipeline on Bitrise, which we’ve setup for our step testing (bitrise.yml can be downloaded here):

1. AVD Manager
2. Git Clone Repository
3. Flutter Install for building our flutter-based app.
4. Bitrise Cache pull
5. Bitrise Cache push
6. Wait For Androind Emulator
7. Script – As our next final Repeato step needs fully ready & configured emulator, we are using commands for building the APK (flutter build apk --split-per-abi) and sending to emulator using adb install (make sure you send/install the apk compatible to your emulator settings x86_64 or arm64 etc…). You might have to configure different step for making the build and sending to device (depending upon your APP). Also make sure before the next step workspace-tests are cloned (if they are in separate repository) – See Useful links section below.
8. Repeato Test Runner Step
9. Deploy to Bitrise.io – Artifacts The Repeato step copies the batch-reports zip file and JUnit XML file into the bitrise upload directory($BITRISE_DEPLOY_DIR). You can use this step to upload the reports into the test artifacts section. You could also use FTP Upload or a Deploy to S3 step for uploading reports.
10. Export test results to Test Reports add-on This Step is to export the results to the Test Reports add-on. The Step creates the required test-info.json file and deploys the test results in the correct directory for export.

Useful links

• Getting started with AVD Manager
• Gradle Runner for build
• Android Build

Simple setup

1. Add a workflow.yml file to your .github/workflow directory.
   A simple push rule (any branch) can be used for workflow action to trigger (You may adjust this according to your requirements)
2. Add a step for launching the emulator
3. Add script for running Repeato CLI:
   adb install demo-app/app-x86_64-release.apk && npx @repeato/cli-testrunner --workspaceDir "${GITHUB_WORKSPACE}/workspace-tests" --batchId 0 --licenseKey ${{ secrets.REPEATO_LICENSE_KEY }} --outputDir "${GITHUB_WORKSPACE}/batch-report"
4. Change the Workspace path, batchId and licenseKey, and other optional switches, according to your needs. For the full set of switches check here.

Example workflow

Here is a demo workflow on Github actions that will help you get going.

Basic configuration: we’ve set the macos-latest for testing purposes. All steps will be performed on this machine. Please use Node.js version 16.

name: Integration with REPEATO CLI

on: [push]

jobs:
  repeato-cli:
    name: Build and run repeato tests using Repeato CLI
    runs-on: macos-latest

    steps:
      # Configure Node.js. Version 16 is currently required
      - uses: actions/setup-node@v3
        with:
          node-version: '16'

      # Check out the app source code
      - uses: actions/checkout@v3

      # Check out the Repeato test workspace. Your tests could also live in the same repository as your app source code. But in this case we decided to have it in it's separate repo.
      - name: Checkout workspace repo
        uses: actions/checkout@v3
        with:
          repository: repeato-qa/demo-workspace
          path: workspace-tests
          clean: false

      # We need Flutter to build the app
      - name: Install Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.0.1'
          channel: 'stable'
      - run: flutter pub get
      - run: flutter build apk --split-per-abi

      # Optional: Add the built apk to the build artifacts
      - name: Upload APK
        uses: actions/upload-artifact@v3
        with:
          name: release-apk
          path: build/app/outputs/flutter-apk/app-x86_64-release.apk

      # Optional: We also commit the built app to the repo. It's saving us some time because we can use the apk in other workflows directly.
      - name: List APKs
        run: ls -la build/app/outputs/flutter-apk/
      
      - name: Move APK to project root
        run: mv build/app/outputs/flutter-apk/*.apk .

      - name: Commit APK
        uses: EndBug/add-and-commit@v9
        with:
          author_name: Github Action
          author_email: stephan@repeato.app
          message: 'Added APK'
          add: '*.apk'
          
      # Finally: Start emulator, install the app and run the tests via npx @repeato/cli-testrunner
      - name: run tests
        uses: reactivecircus/android-emulator-runner@d7b53ddc6e44254e1f4cf4a6ad67345837027a66
        with:
          api-level: 29
          arch: x86_64
          force-avd-creation: false
          emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
          disable-animations: true
          script: adb install ./app-x86_64-release.apk && npx @repeato/cli-testrunner --workspaceDir "${GITHUB_WORKSPACE}/workspace-tests" --batchId 0 --licenseKey ${{ secrets.REPEATO_LICENSE_KEY }} --outputDir "${GITHUB_WORKSPACE}/batch-report"

      # Optional: Upload the reports to some bucket so a link to the batch run report can be easily shared
      # For AWS S3 You must enable enable ACLs object writing for bucket.
      # Don't put your access keys in here. Use Github secrets for storing & accessing the keys.
      # You might want to use 'if: always()' in this step and following ones, so the report is also uploaded if your test batch run failed.
#      - name: Upload report to S3
#        uses: shallwefootball/s3-upload-action@master
#        with:
#          aws_key_id: $AWS_ACCESS_KEY
#          aws_secret_access_key: $AWS_ACCESS_SECRET_KEY
#          aws_bucket: $AWS_BUCKET
#          source_dir: batch-report