Currently, we offer two products:
We currently don’t offer any cloud services for running your tests.
However, you can use our test runner to run tests on third-party services such as Github Actions, Bitrise, Codemagic or CircleCI.
Since those services are only supporting virtual devices, we recommend also recording your tests on the same virtual devices. This way you will get the best test stability and performance.
Even though Repeato CLI will run on pretty much any server, we currently offer 3 setup guides (more to come):
Add a “Repeato Test Runner” step to your workflow in the Bitrise Workflow Editor. You can also run the “Repeato Test Runner” step directly with Bitrise CLI.
The step takes care of checking the system requirements as well as installing and starting the headless Repeato CLI test runner. After running the tests, it sets the environment variables that can be used in consecutive steps.
Please make sure AVD or IOS emulator is configured properly. Further, your Android/iOS app build must be installed on the emulator before running the test batch. We recommend to run the tests on the same device / emulator you originally used for creating them. This way you will get the most stability and performance.
For iOS Simulator there isn’t any step, that starts the simulator, as the xcodebuild test
command for iOS Testing step boots the simulator by default.
Key | Description | Flags | Default |
---|---|---|---|
repeato_cli_version | Set the repeato CLI version compatible to your workspace tests. | required | latest |
workspace_path | Repeato test runner need workspace path for setting up the workspace before executing batch. | required | $BITRISE_SOURCE_DIR |
batch_id | Set batch id for the tests execution. | required | 0 |
license_key | Set license key for the tests execution. | required | none |
Environment Variable | Description |
---|---|
REPEATO_REPORT | Repeato Batch Report Zip File |
REPEATO_JUNIT_REPORT | Repeato JUnit XML File |
The emulator needs some time to boot up. The earlier you place the step in your workflow, the more tasks, such as cloning or caching, you can complete in your workflow before the emulator starts working. We recommend that you also add Wait for Android emulator Step (in the case of android) to your workflow as it acts as a shield preventing the AVD manager to kick in too early. Make sure you add the Wait for Android emulator Step BEFORE the Repeato Test Runner Step
so our step can use AVD Manager.
Here is the our demo pipeline on Bitrise, which we’ve setup for our step testing (bitrise.yml can be downloaded here):
flutter build apk --split-per-abi
) and sending to emulator using adb install
(make sure you send/install the apk compatible to your emulator settings x86_64 or arm64
etc…). You might have to configure different step for making the build and sending to device (depending upon your APP). Also make sure before the next step workspace-tests are cloned (if they are in separate repository) – See Useful links
section below.batch-reports
zip file and JUnit XML file into the bitrise upload directory($BITRISE_DEPLOY_DIR
). You can use this step to upload the reports into the test artifacts section. You could also use FTP Upload or a Deploy to S3 step for uploading reports.
push
rule (any branch) can be used for workflow action to trigger (You may adjust this according to your requirements)adb install demo-app/app-x86_64-release.apk && npx @repeato/cli-testrunner --workspaceDir "${GITHUB_WORKSPACE}/workspace-tests" --batchId 0 --licenseKey ${{ secrets.REPEATO_LICENSE_KEY }} --outputDir "${GITHUB_WORKSPACE}/batch-report"
Here is a demo workflow on Github actions that will help you get going.
Basic configuration: we’ve set the macos-latest
for testing purposes. All steps will be performed on this machine. Please use Node.js version 16.
name: Integration with REPEATO CLI
on: [push]
jobs:
repeato-cli:
name: Build and run repeato tests using Repeato CLI
runs-on: macos-latest
steps:
# Configure Node.js. Version 16 is currently required
- uses: actions/setup-node@v3
with:
node-version: '16'
# Check out the app source code
- uses: actions/checkout@v3
# Check out the Repeato test workspace. Your tests could also live in the same repository as your app source code. But in this case we decided to have it in it's separate repo.
- name: Checkout workspace repo
uses: actions/checkout@v3
with:
repository: repeato-qa/demo-workspace
path: workspace-tests
clean: false
# We need Flutter to build the app
- name: Install Flutter
uses: subosito/flutter-action@v2
with:
flutter-version: '3.0.1'
channel: 'stable'
- run: flutter pub get
- run: flutter build apk --split-per-abi
# Optional: Add the built apk to the build artifacts
- name: Upload APK
uses: actions/upload-artifact@v3
with:
name: release-apk
path: build/app/outputs/flutter-apk/app-x86_64-release.apk
# Optional: We also commit the built app to the repo. It's saving us some time because we can use the apk in other workflows directly.
- name: List APKs
run: ls -la build/app/outputs/flutter-apk/
- name: Move APK to project root
run: mv build/app/outputs/flutter-apk/*.apk .
- name: Commit APK
uses: EndBug/add-and-commit@v9
with:
author_name: Github Action
author_email: stephan@repeato.app
message: 'Added APK'
add: '*.apk'
# Finally: Start emulator, install the app and run the tests via npx @repeato/cli-testrunner
- name: run tests
uses: reactivecircus/android-emulator-runner@d7b53ddc6e44254e1f4cf4a6ad67345837027a66
with:
api-level: 29
arch: x86_64
force-avd-creation: false
emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none
disable-animations: true
script: adb install ./app-x86_64-release.apk && npx @repeato/cli-testrunner --workspaceDir "${GITHUB_WORKSPACE}/workspace-tests" --batchId 0 --licenseKey ${{ secrets.REPEATO_LICENSE_KEY }} --outputDir "${GITHUB_WORKSPACE}/batch-report"
# Optional: Upload the reports to some bucket so a link to the batch run report can be easily shared
# For AWS S3 You must enable enable ACLs object writing for bucket.
# Don't put your access keys in here. Use Github secrets for storing & accessing the keys.
# You might want to use 'if: always()' in this step and following ones, so the report is also uploaded if your test batch run failed.
# - name: Upload report to S3
# uses: shallwefootball/s3-upload-action@master
# with:
# aws_key_id: $AWS_ACCESS_KEY
# aws_secret_access_key: $AWS_ACCESS_SECRET_KEY
# aws_bucket: $AWS_BUCKET
# source_dir: batch-report