Warning The bug reported by this demo has been fixed. The fix was simple and is described in cenfun/monocart-reporter#29 and implemented in the
fixbranch. Use this compare view to see the changed files.
Note This repo is still useful as it shows two ways to get code coverage working on playwright, one with
instanbuland one usingV8+monocart-reporter. For those interested in playwright and code coverage you might find it helpful to read this comment which explains my journey into getting code coverage working on playwright.
This is a demo repo to check a potential bug in monocart-reporter related with code coverage tracked in cenfun/monocart-reporter#29.
The repo consists of an Angular app running playwright tests. The aim is to get accurate code coverage from the playwright tests that run the Angular app via playwright's webServer.
See here for a description of the bug.
Run npm run start from the root of the repo. This command is not necessary to reproduce the bug but might be useful to know in case you want to run the app.
Run npm run test-istanbul from the root of the repo. This command will:
- cleans test results and code coverage directories to make sure we are always on a fresh state.
- updates the
angular.jsonfile that is at the root of the repo so that the Angular app will be instrumented using istanbul. - runs the playwright tests at
/tests/babel-istanbul. This uses a playwright.config.ts and base fixture configured to collect code coverage from istanbul. - executes the nyc command to generate an html report and lcov file at
/coverage.
After running npm run test-istanbul you can open the code coverage report by running npm run open-istanbul-coverage.
Run npm run test-monocart from the root of the repo. This command will:
- cleans test results and code coverage directories to make sure we are always on a fresh state.
- updates the
angular.jsonfile that is at the root of the repo. - runs the playwright tests at
/tests/monocart. This uses a playwright.config.ts and example.spec.ts test configured to collect code coverage using playwright's coverage api (V8) and then produce a report usingmonocart-reporter.
After running npm run test-monocart you can open the code coverage report by running npm run open-monocart-coverage.
When instrumenting the playwright tests using istanbul via the babel-plugin-istanbul plugin the code coverage shows accurately whilst when using V8 and monocart-reporter some lines that should be covered show as uncovered even though there's an indication those lines have been executed X number of times.
please be patient, the npm commands to execute the tests take about 20s to complete on my machine.
- clone the repo.
- go to the root of the repo.
- run
npm run test-istanbul. - run
npm run test-monocart. - run
npm run open-istanbul-coverageand on the coverage report which opened in the browser go to the code coverage for thefind-institution.view-model.tsfile which is at/app/find-institution/find-institution.view-model.ts.html. - run
npm run open-monocart-coverageand on the coverage report which opened in the browser go to the code coverage for thefind-institution.view-model.tsfile which is atwebpack:src/app/find-institution/find-institution.view-model.ts.html. - compare the code coverages for the
find-institution.view-model.tsin the two reports. The istanbul code coverage shows accurate coverage whilst the monocart code coverage does not. What is curious is that even in uncovered lines, the monocart report still shows they were executed X number of times.
The images below show a part of the code coverage for the find-institution.view-model.ts file which show the problem
- Istanbul code coverage report for
find-institution.view-model.ts:
- Monocart code coverage report with V8 for
find-institution.view-model.ts:
- The step to update the
angular.jsonfile in thenpm run test-istanbulandnpm run test-monocartcommands was a way I found to to swap parts of theangular.jsonconfiguration which is different when running the Angular app with istanbul instrumentation or without.- Without istanbul instrumentation, there's nothing that needs to be done to the default
angular.jsonfile you get when creating an Angular app. - With istanbul istrumentation, you have to be able to extend angular's webpack configuration so that it runs the babel-plugin-istanbul plugin which will instrument your code. I followed the idea described in Use Istanbul coverage collection with Playwright Test and applied it to an angular project. To extend angular's webpack configuration I used ngx-build-plus.
- Without istanbul instrumentation, there's nothing that needs to be done to the default
- The code of the app has been stripped down to a minimum to reproduce the issue with code coverage so it might not make much sense. The code itself is irrelevant, what matters is understanding why the code coverage shows differently.