The one-and-only known GitHub action (and solution), that allows you to finish your PlayWright tests in your chosen x minutes, while using optimal minimum number of GitHub runners. **
** At the time of writing, there are no known other solutions (paid or open source), that can do this.
-
🚀 Faster than Playwright Sharding: ✅
-
Smart load balancing based on execution time and not just test count: We create "bundle of tests and runners" based on:
- The time each test takes.
- The total exepected time for run provided by user.
- The cores availablity of runners.
This makes make our total run time predictable and fast.
-
Distributed Run: The number of bundles translates to number of runners.
-
Parallel Run: The number of cores translates to number of workers (where workers = cores/2).
-
-
↗️ ↘️ Dynamic sharding over Playwrights hard coded Sharding: ✅- Although this is not directly a Playwright sharder shortcoming but since not everyone is an expert with GitHub actions and the playwright sharding github example hard codes the number of shards in the workflow file, most teams end up using the example as-is in their projects and thus results into a inefficient hard coded runner strategy.
Teams often increase runners to finish tests faster but when they do maintenance or add a few new tests, they end up spinning all those runners, resulting into waste and under utilisation of runners.
-
💸 Cheaper than Playwright Sharding: ✅
- Since we dynamically scale runners up and down to always create only the bare minimum runners required to do the job, we avoid waste in terms of CI minutes. For comparision, with playwright sharding teams end up hard coding more runners to bring down execution time and as a side affect increase cost to the company.
-
🌿 Greener than Playwright Sharding: ✅
- Since we always only create the exact amount of runners we need to do the job (no less, no more), and since each runner is optimially utilised to finish all runners at approx same time, this is also a very efficient and thus greener alternative to Playwright Sharding.
This article explains, in detail, the need for this action and the problem it solves.
There are 3 main steps involved:
-
Add a
pre-commithook file as shown here.- This will run
--only-changedtests on local commits.
- This will run
-
Copy state-reporter.js file and put it in the root repository.
- This will create a
[state.json](https://github.com/PramodKumarYadav/playwright-sandbox/blob/main/state.json)file that contains the mapping of test path and the time it took to run (in ms).
- This will create a
-
Update playwright.config.ts file reporters to include this reporter as shown below.
reporter: [["list"], ["html"], ["github"], ["./state-reporter.js"]], -
Add a
post-commithook file as shown here.- This will automatically add
[state.json](https://github.com/PramodKumarYadav/playwright-sandbox/blob/main/state.json)file to the branch.
- This will automatically add
-
Add a reusable workflow that can take inputs from user to run playwright commands and finish tests in x mins.
-
Add a example trigger workflow that shows how to use the reusable workflow to run desired tests.
Step2: Run tests based on the defined triggers (push to main, pull_request to main, schedule etc all)
- Create an event that triggers the workflow and verify if the tests finish in approx same projected time.
- For now, you can use the trigger workflow you copied above to trigger and run these tests. Fork the repo and push something on your main branch to trigger tests.
- If you find any issues, use the issues page to raise them.
-
This action is made to work with playwright option
fully parallel = true. The action is not meant to deal with tests run inserialordefaultmode and thus can have side effects if your tests are not running fully parallel. This is intended to be addressed in one of future releases. -
Do not use sharding related commands in the input playwright command to run; since this solution is meant to overcome the flaws of sharding. Using sharding again would introduce those short comings again.
-
If you are using custom powerful GitHub runners, use the same custom runner type for job that evaluates "RunWright" then what you would use in subseque AEFA nt job for running tests.
inputs:
total-run-time-in-mins:
description: 'desired-total-test-run-time-in-mins'
required: true
pw-command-to-execute:
description: 'playwright command to run tests'
required: true
outputs:
dynamic-matrix:
description: "dynamic matrix to use"
value: ${{ steps.set-matrix.outputs.dynamic_matrix }}
test-load-distribution-json:
description: "test load distribution json"
value: ${{ steps.calculate-required-runners.outputs.test_load_json }}
recommended-workers:
description: 'optimal number of workers to run tests'
value: ${{ steps.get-number-of-cpu-cores-to-decide-on-worker-count.outputs.RECOMMENDED_WORKERS }}
Follow the instructions in Getting Started section that shows how to use this action. Other than that, below are some common examples:
- run only new or updated tests in a pull request
- run all tests on a push to main
- run selected tests on demand
To create and push new tags (releases) of this action:
pramodyadav@Pramods-Laptop runwright % git tag -a -m "add your message here" v1
pramodyadav@Pramods-Laptop runwright % git push --follow-tags
Below are some tests to verify for some edge case scenarios and validate that the action works as expected.
all 1.5k tests finished in less than 2 mins
all 3k tests finished in approx 2 mins
- It could be a good idea to generate the
[state.json](https://github.com/PramodKumarYadav/playwright-sandbox/blob/main/state.json)file from scratch every few days or weeks to avoid having redundant test path and names.
-
Add option to deal with
fullyParallel=false. This should cover the below scenarios:- When
fullyParallel=false. - When
fullyParallel=truebut there are test files that containstest.describe.configure({ mode: 'serial' });ortest.describe.configure({ mode: 'default' });
- When
-
Add option for when a user doesn't want to limit by time but want to limit the maximum runners to use.