An introduction to mutation testing
How code coverage of 100% could mean less than 60% is tested.
No time to run the example yourself? Don't worry, we did it for you. Open it right in your browser:
The RoboBar is a small application to demo mutation testing. It actually has a fair amount of unit tests. When we wrote this application, we didn't even try our best to write bad tests. We just focussed on code coverage and didn't practice Test Driven Development. It turns out it's really easy to write bad tests or forget a few important test cases. The RoboBar even has a fairly large bug. Finding it is pretty easy using the mutation report. Why don't you give it a try? 😁
Note: The RoboBar is developed using native web components without a frontend framework. This is done on purpose to keep this example as accessible as possible, as well as to keep the maintenance burden low.
- Install git
- Install nodejs
- Open command prompt and clone this repository:
git clone https://github.com/stryker-mutator/robobar-example
- Change directory into the robobar and install the dependencies.
cd robobar-example npm install - Run tests with npm. This will generate a code coverage report.
npm test - Review the 100% code coverage score. Open up the code coverage report located in the
reports/coverage/lcov-reportdirectory. - Run mutation testing with Stryker
npm run test:mutation
- Review the less than 60% mutation score. Open up the mutation report located in the
reports/mutationdirectory. - Run the website with
npm start. Can you find the bug?
If you want to install stryker yourself, step back in history using git:
git checkout pre-stryker
npm installAfter that you can install stryker for yourself:
npm i -D @stryker-mutator/core
npx stryker initChoose the following options in the questionnaire:
- Are you using one of these frameworks?
None/other - Which test runner do you want to use?
jest - Reporters:
html,clear-text,progress - Which package manager do you want to use?:
npm - What file type do you want for your config file?:
json
After the plugins are installed, open the stryker.conf.json file and make the following change:
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"_comment": "This config was generated using 'stryker init'. Please take a look at: https://stryker-mutator.io/docs/stryker-js/configuration/ for more information",
"packageManager": "npm",
"reporters": [
"html",
"clear-text",
"progress",
"dashboard"
],
"testRunner": "jest",
- "coverageAnalysis": "perTest"
+ "coverageAnalysis": "perTest",
+ "testRunnerNodeArgs": ["--experimental-vm-modules"],
+ "jest": {
+ "configFile": "jest.config.cjs"
+ }
}(this is needed because we're using jest with ECMAScript modules)
After the plugins are installed, try it out:
npx stryker run