Flank is a massively parallel Android and iOS test runner for Firebase Test Lab.
- Install Oracle JDK 8
- JDK 9 or later will not work
- Use JetBrains Toolbox to install
IntelliJ IDEA Community - Clone the repo
git clone --recursive https://github.com/TestArmada/flank.gitgit submodule update --init --recursiveupdates the submodules
- Open
test_runner/build.gradle.ktswithIntelliJ IDEA Community
| Available now | 2019 |
|---|---|
| Test sharding | Client/Server refactor |
| Cost reporting | |
| Stability testing | |
| HTML report | |
| JUnit XML report | |
| Smart Flank |
| Exit code | Description |
|---|---|
| 0 | All tests passed |
| 1 | At least one test failed or inconclusive and all matrices finished. |
| 2 | At least one matrix not finished, usually a FTL error. |
app, test, and xctestrun-file support ~, environment variables, and globs (*, **) when resolving paths
Run test_runner/flank.ios.yml with flank to verify iOS execution is working.
cd test_runner/./gradlew clean assemble fatJarjava -jar ./build/libs/flank-*.jar firebase test ios run
# gcloud args match the official gcloud cli
# https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test/ios/run
gcloud:
# results-bucket: tmp_flank
record-video: true
timeout: 30m
async: false
# project: delta-essence-114723
# results-history-name: ios-history
# test and xctestrun-file are the only required args
test: ./src/test/kotlin/ftl/fixtures/tmp/EarlGreyExample.zip
xctestrun-file: ./src/test/kotlin/ftl/fixtures/tmp/EarlGreyExampleSwiftTests_iphoneos12.1-arm64e.xctestrun
xcode-version: 9.2
device:
- model: iphone8
version: 11.2
locale: en
orientation: portrait
# The number of times to retry failed tests. Default is 0. Max is 10.
flaky-test-attempts: 0
flank:
# test shards - the amount of groups to split the test suite into
# set to -1 to use one shard per test.
testShards: 1
# repeat tests - the amount of times to run the tests.
# 1 runs the tests once. 10 runs all the tests 10x
repeatTests: 1
# always run - these tests are inserted at the beginning of every shard
# useful if you need to grant permissions or login before other tests run
test-targets-always-run:
- a/testGrantPermissions
# test targets - a list of tests to run. omit to run all tests.
test-targets:
- b/testBasicSelection
# regex is matched against bucket paths, for example: 2019-01-09_00:18:07.314000_hCMY/shard_0/EarlGreyExampleSwiftTests_iphoneos12.1-arm64e.xctestrun
files-to-download:
- .*\.png$Run test_runner/flank.yml with flank to verify Android execution is working.
cd test_runner/./gradlew clean assemble fatJarjava -jar ./build/libs/flank-*.jar firebase test android run
# gcloud args match the official gcloud cli
# https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
gcloud:
results-bucket: tmp_flank
record-video: true
timeout: 30m
async: false
project: delta-essence-114723
results-history-name: android-history
# test and app are the only required args
app: ../test_app/apks/app-debug.apk
test: ../test_app/apks/app-debug-androidTest.apk
auto-google-login: true
use-orchestrator: true
environment-variables:
clearPackageData: true
directories-to-pull:
- /sdcard/screenshots
performance-metrics: true
test-targets:
# supported are class, notClass, size, annotation, notAnnotation, package, notPackage, testFile, notTestFile
# as described in https://developer.android.com/reference/android/support/test/runner/AndroidJUnitRunner
- class com.example.app.ExampleUiTest#testPasses
device:
- model: NexusLowRes
version: 28
# The number of times to retry failed tests. Default is 0. Max is 10.
flaky-test-attempts: 0
flank:
# test shards - the amount of groups to split the test suite into
# set to -1 to use one shard per test.
testShards: 1
# repeat tests - the amount of times to run the tests.
# 1 runs the tests once. 10 runs all the tests 10x
repeatTests: 1
# always run - these tests are inserted at the beginning of every shard
# useful if you need to grant permissions or login before other tests run
test-targets-always-run:
- class com.example.app.ExampleUiTest#testPasses
# regex is matched against bucket paths, for example: 2019-01-09_00:13:06.106000_YCKl/shard_0/NexusLowRes-28-en-portrait/bugreport.txt
files-to-download:
- .*\.mp4$Download Flank from GitHub releases.
Stable
wget --quiet https://github.com/TestArmada/flank/releases/download/v3.0.0/flank.jar -O ./flank.jar
java -jar ./flank.jar android run
Snapshot (published after every commit)
wget --quiet https://github.com/TestArmada/flank/releases/download/flank_snapshot/flank.jar -O ./flank.jar
java -jar ./flank.jar android run
In CI, it may be useful to generate the file via a shell script:
cat << 'EOF' > ./flank.yml
gcloud:
app: ../../test_app/apks/app-debug.apk
test: ../../test_app/apks/app-debug-androidTest.apk
EOF
Fladle is a Gradle plugin for Flank that provides DSL configuration and task based execution.
Run flank auth login. Flank will save the credential to ~/.flank. Google account authentication allows each person
to have a unique non-shared credential. A service account is still recommended for CI.
Follow the test lab docs to create a service account.
- Save the credential to
$HOME/.config/gcloud/application_default_credentials.jsonor setGOOGLE_APPLICATION_CREDENTIALSwhen using a custom path. - Set the project id in flank.yml or set the
GOOGLE_CLOUD_PROJECTenvironment variable.
For continuous integration, base64 encode the credential as GCLOUD_KEY. Then write the file using a shell script. Note that gcloud CLI does not need to be installed. Flank works without any dependency on gcloud CLI.
Encode JSON locally.
base64 -i "$HOME/.config/gcloud/application_default_credentials.json" | pbcopyThen in CI decode the JSON.
GCLOUD_DIR="$HOME/.config/gcloud/"
mkdir -p "$GCLOUD_DIR"
echo "$GCLOUD_KEY" | base64 --decode > "$GCLOUD_DIR/application_default_credentials.json"flank.yml is compatible with the gcloud CLI.
gcloud firebase test android run flank.yml:gcloudgcloud alpha firebase test ios run flank.ios.yml:gcloud
NOTE: You will need to activate gcloud's service account for the above commands to work.
Use the doctor command to check for errors in the YAML.
flank firebase test android doctorflank firebase test ios doctor
Flank supports printing the current version.
$ flank -v
v3.0-SNAPSHOTAccess Not Configured. Cloud Tool Results API has not been used in project 764086051850 before or it is disabled.
This error means authentication hasn't been setup properly. See Authenticate with a service account in this readme.
How do I use Flank without typing long commands?
Add Flank's bash helper folder to your $PATH environment variable. This will allow you to call the shell scripts in that helper folder from anywhere.
With the flank shell script, you can use flank instead of java -jar flank.jar. Examples:
flank android runflank ios run
With the update_flank.sh shell script, you can rebuild flank.jar.
Symbol is declared in module 'java.xml' which does not export package 'com.sun.org.apache.xerces.internal.dom'
Make sure you're using JDK 8 to compile Flank.
Test run failed to complete. Expected 786 tests, received 660
Try setting use-orchestrator: false. Parameterized tests are not compatible with orchestrator. Flank uses orchestrator by default on Android.