Running tests as part of your Continuous Integration

Currently, we offer two products:

• Repeato Studio: A desktop app that allows you to easily create and edit tests and test batches ➡ Documentation
• Repeato CLI: A headless Node.js test runner application, that will enable you to run test batches on a server. ➡ Documentation

We currently don’t offer any cloud services for running your tests.
However, you can use our test runner to run tests on third-party services such as Github Actions, Bitrise, Codemagic or CircleCI.

Since those services are only supporting virtual devices, we recommend also recording your tests on the same virtual devices. This way you will get the best test stability and performance.

Even though Repeato CLI will run on pretty much any server, we currently offer 4 setup guides (more to come):

• • Bitrise
  • Github Actions
  • Codemagic
  • Appcircle
  • Your own server

Setup

Add a “Repeato Test Runner” step to your workflow in the Bitrise Workflow Editor. You can also run the “Repeato Test Runner” step directly with Bitrise CLI.

The step takes care of checking the system requirements as well as installing and starting the headless Repeato CLI test runner. After running the tests, it sets the environment variables that can be used in consecutive steps.

Setup Steps

1. Add the Repeato Test Runner Step to your Workflow as one of the Steps in your Workflow.
2. Make sure your test workspace is checked out.
3. Set the Workspace Path to point to the repeato workspace directory.
4. Set the Batch Id. The batch ID of the test batch you want to execute.
5. Set the Repeato Access Token (can be created here).

Please make sure the AVD or IOS emulator is configured properly. Further, your Android/iOS app build must be installed on the emulator before running the test batch. We recommend running the tests on the same device / emulator you originally used for creating them. This way you will get the most stability and performance.

For iOS Simulators there isn’t any step, that starts the simulator, as the xcodebuild test command for the iOS testing step boots the simulator by default.

Inputs

Outputs

Tips & Troubleshooting

The emulator needs some time to boot up. The earlier you place the step in your workflow, the more tasks, such as cloning or caching, you can complete in your workflow before the emulator starts working. We recommend that you also add Wait for Android emulator Step (in the case of android) to your workflow as it acts as a shield preventing the AVD manager to kick in too early. Make sure you add the Wait for Android emulator Step BEFORE the Repeato Test Runner Step so our step can use AVD Manager.

Demo Pipeline (Android / Flutter)

Here is the our demo pipeline on Bitrise, which we’ve setup for our step testing (bitrise.yml can be downloaded here):

1. AVD Manager
2. Git Clone Repository
3. Flutter Install for building our flutter-based app.
4. Bitrise Cache pull
5. Bitrise Cache push
6. Wait For Androind Emulator
7. Script – As our next final Repeato step needs fully ready & configured emulator, we are using commands for building the APK (flutter build apk --split-per-abi) and sending to emulator using adb install (make sure you send/install the apk compatible to your emulator settings x86_64 or arm64 etc…). You might have to configure different step for making the build and sending to device (depending upon your APP). Also make sure before the next step workspace-tests are cloned (if they are in separate repository) – See Useful links section below.
8. Repeato Test Runner Step
9. Deploy to Bitrise.io – Artifacts The Repeato step copies the batch-reports zip file and JUnit XML file into the bitrise upload directory($BITRISE_DEPLOY_DIR). You can use this step to upload the reports into the test artifacts section. You could also use FTP Upload or a Deploy to S3 step for uploading reports.
10. Export test results to Test Reports add-on This Step is to export the results to the Test Reports add-on. The Step creates the required test-info.json file and deploys the test results in the correct directory for export.

Useful links

• Getting started with AVD Manager
• Gradle Runner for build
• Android Build

Setup Steps

1. Add a workflow.yml file to your .github/workflow directory.
   A simple push rule (any branch) can be used for workflow action to trigger (You may adjust this according to your requirements)
2. Add a step for launching the emulator
3. Add script for running Repeato CLI:
   adb install demo-app/app-x86_64-release.apk && npx @repeato/cli-testrunner --workspaceDir "${GITHUB_WORKSPACE}/workspace-tests" --batchId 0 --accessToken ${{ secrets.REPEATO_ACCESS_TOKEN }} --outputDir "${GITHUB_WORKSPACE}/batch-report"
4. Change the Workspace path, batchId and accessToken, and other optional switches, according to your needs. For the full set of switches check here.

Example workflow

Here is a demo workflow on Github actions that will help you get going.

Basic configuration: we’ve set the macos-latest for testing purposes. All steps will be performed on this machine. Please use Node.js version 16.

name: Integration with REPEATO CLI

on: [push]

jobs:
  repeato-cli:
    name: Build and run repeato tests using Repeato CLI
    runs-on: macos-latest

    steps:
      # Configure Node.js. Version 16 is currently required
      - uses: actions/setup-node@v3
        with:
          node-version: '16'

      # Check out the app source code
      - uses: actions/checkout@v3

      # Check out the Repeato test workspace. Your tests could also live in the same repository as your app source code. But in this case we decided to have it in it's separate repo.
      - name: Checkout workspace repo
        uses: actions/checkout@v3
        with:
          repository: repeato-qa/demo-workspace
          path: workspace-tests
          clean: false

      # We need Flutter to build the app
      - name: Install Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.0.1'
          channel: 'stable'
      - run: flutter pub get
      - run: flutter build apk --split-per-abi

      # Optional: Add the built apk to the build artifacts
      - name: Upload APK
        uses: actions/upload-artifact@v3
        with:
          name: release-apk
          path: build/app/outputs/flutter-apk/app-x86_64-release.apk

      # Optional: We also commit the built app to the repo. It's saving us some time because we can use the apk in other workflows directly.
      - name: List APKs
        run: ls -la build/app/outputs/flutter-apk/
      
      - name: Move APK to project root
        run: mv build/app/outputs/flutter-apk/*.apk .

      - name: Commit APK
        uses: EndBug/add-and-commit@v9
        with:
          author_name: Github Action
          author_email: stephan@repeato.app
          message: 'Added APK'
          add: '*.apk'
          
      # Finally: Start emulator, install the app and run the tests via npx @repeato/cli-testrunner
      - name: run tests
        uses: reactivecircus/android-emulator-runner@d7b53ddc6e44254e1f4cf4a6ad67345837027a66
        with:
          api-level: 29
          arch: x86_64
          force-avd-creation: false
          emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
          disable-animations: true
          script: adb install ./app-x86_64-release.apk && npx @repeato/cli-testrunner --workspaceDir "${GITHUB_WORKSPACE}/workspace-tests" --batchId 0 --accessToken ${{ secrets.REPEATO_ACCESS_TOKEN }} --outputDir "${GITHUB_WORKSPACE}/batch-report"

      # Optional: Upload the reports to some bucket so a link to the batch run report can be easily shared
      # For AWS S3 You must enable enable ACLs object writing for bucket.
      # Don't put your access keys in here. Use Github secrets for storing & accessing the keys.
      # You might want to use 'if: always()' in this step and following ones, so the report is also uploaded if your test batch run failed.
#      - name: Upload report to S3
#        uses: shallwefootball/s3-upload-action@master
#        with:
#          aws_key_id: $AWS_ACCESS_KEY
#          aws_secret_access_key: $AWS_ACCESS_SECRET_KEY
#          aws_bucket: $AWS_BUCKET
#          source_dir: batch-report

Codemagic is a cloud-based continuous integration and delivery (CI/CD) service that enables developers to build, test, and deploy their mobile apps to multiple platforms. It is a part of the Flutter ecosystem and supports both Flutter and React Native apps. Codemagic can help developers automate many of the tasks involved in building mobile apps, including testing, building, and deploying to app stores.

Setup Steps

Here are the steps to follow:

1. Go to www.codemagic.io, and create an account if you don’t already have one
2. Create a new Codemagic project and select your Git repository.
3. Choose the platform you want to build for (iOS, Android, or both).
4. Configure the build settings, including the build type, version number, and bundle identifier.
5. Add any required environment variables or secrets.
6. Test your workflow: Create a build and wait for it to complete.

Example Workflows

This is what the the workflow does, as soon as you push changes to your Git repository:

1. Build Workflow:
   1. Environment setup
   2. Flutter setup/installation
   3. Flutter APK build
   4. Add the generated APK build in artifacts
2. Test Run Workflow
   1. Setup environment with node and flutter version
   2. Launch emulator
   3. Install app in emulator
   4. Run Repeato tests
   5. Publish the Repeato test reports to S3 bucket
   6. Optionally send the notification on slack on build complete
   7. If the build fails, you will receive a notification and can investigate the issue.

workflows:
  build-workflow:
    name: Build Workflow
    instance_type: mac_pro
    max_build_duration: 120
    environment:
      node: 16.19.0
      flutter: 3.0.1
      groups:
        - standard
    scripts:
      - name: Set up local.properties
        script: |
          #!/usr/bin/env zsh 
          echo "flutter.sdk=$HOME/programs/flutter" > "$CM_BUILD_DIR/android/local.properties"
      - name: Get Flutter packages
        script: flutter packages pub get
      - name: Build apk with Flutter
        script: flutter build apk
    publishing:
      scripts:
        - name: List exported files
          script: | 
            ls -la $CM_EXPORT_DIR
    artifacts:
      - build/**/outputs/apk/**/*.apk
  test-android-workflow:
    name: Test Workflow
    instance_type: mac_pro
    max_build_duration: 120
    environment:
      node: 16.19.0
      flutter: 3.0.1
      groups:
        - standard
    scripts:
      - name: Set up local.properties
        script: |
          #!/usr/bin/env zsh 
          echo "flutter.sdk=$HOME/programs/flutter" > "$CM_BUILD_DIR/android/local.properties"      
      - name: Launch emulator
        script: |
          emulator @emulator &  # Use `emulator -list-avds` to list all emulators
          adb wait-for-device # wait for the emulator to finish loading
      - name: Install app
        script: |
          adb install app.apk
      - name: Wait for 20 seconds
        script: sleep $((20*1))
      - name: Run Repeato tests
        script: |
          npx @repeato/cli-testrunner \
          --workspaceDir "demo-workspace" \
          --logLevel "DEBUG" \
          --batchId 0 \
          --accessToken "$REPEATO_ACCESS_TOKEN" \
          --waitDurationBetweenSteps 2000 \
          --timeoutFactor 1.5 \
          --outputDir "$CM_EXPORT_DIR/batch-report" 
    publishing:
      scripts:
        - name: List exported files
          script: | 
            ls -la $CM_EXPORT_DIR
            ls -la $CM_EXPORT_DIR/batch-report
        - name: Publish to S3 bucket
          script: | 
            sudo pip3 install awscli --upgrade
            aws s3 sync "$CM_EXPORT_DIR/batch-report" s3://repeato-batch-result-demo/batch-report
            echo "open report here: https://repeato-batch-result-demo.s3.eu-central-1.amazonaws.com/batch-report/index.html"
      slack:
        channel: '#dev'
        notify_on_build_start: false # To receive a notification when a build starts
        notify:
          success: true 
          failure: true 

* Repeato CLI is currently only officially supported on Codemagics’ Mac machines.

Appcircle is a cloud-based platform for continuous integration and continuous delivery (CI/CD) that helps automate the build, test, and deployment processes of mobile applications. It offers a comprehensive set of tools and features to automate and streamline the build, test, and deployment processes of mobile applications.

Additionally, with Appcircle self-hosted, you can host the CI/CD infrastructure on your own servers or in a private cloud environment, providing you with complete control and ownership over the infrastructure.

Prerequisites

Before you begin, make sure you have the following:

• An active account on Appcircle and Repeato
• Access to your project in both Appcircle and Repeato
• Your mobile application source code stored in a version control system (e.g., GitHub, Bitbucket, GitLab)

Configuring the CI/CD Workflow

1. In Appcircle, go to your project’s build page and select the “Workflows” tab.
2. To integrate Repeato into the workflow, click “Manage Workflow” and add “Repeato Test Runner” from the list of available steps.
3. Add the “Xcodebuild for iOS Simulator” step for iOS or the “Wait for Android Emulator” for Android, before the “Repeato Test Runner” step.
4. Add the “Test Reports” step after the “Repeato Test Runner” step. This will allow the test results to be displayed on the build page.
5. Configure the Repeato step by providing the necessary parameters such as “Batch ID”, “Access Token” and “CLI Version” with the information you get from Repeato.
6. Configure the remaining fields, “Workspace Path” and “Log Level,” with the necessary information.
7. Save the workflow configuration.

Important Note: For the “Test Reports” step, make sure to enable the “Always run this step even if the previous steps fail” option. This ensures that you will receive reports for both successful and failed tests.

Running the CI/CD Workflow

1. Trigger the workflow manually or configure automatic triggers, such as on commit or merge events, depending on your preferences.
2. Appcircle will initiate the workflow, which includes the build, test, and deployment steps.
3. The Repeato step in the workflow will execute the specified test cases and workflows using the Repeato API.
4. Appcircle will provide detailed logs and reports for each step, including the Repeato test results.

Get even more detailed information and tips on the Appcircle website