nyc: Merging coverage files doesn't produce accurate cumulative result
I execute the following nyc commands, after generating individual coverage reports on the same codebase using different test executions run by Jest and Cypress.
nyc merge coverage-jest combined/coverage-jest.json
nyc merge coverage-cypress combined/coverage-cypress.json
nyc report combined --reporter html --reporter text
Expected Behavior
I’d expect this to show a correct cumulative coverage report of .ts and .html files.
Observed Behavior
- Coverage of .html files is not shown in merged report, while they are show in individial report.
- Coverage of some .ts files is shown correctly, but for others a
Cannot read property 'locations' of undefined
TypeError
is shown.
Troubleshooting steps
- still occurring when I put
cache: false
in my nyc config
Environment Information
System:
OS: Windows 10 10.0.18363
CPU: (8) x64 Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
Memory: 7.97 GB / 31.88 GB
Binaries:
Node: 13.6.0 - C:\Program Files\nodejs\node.EXE
Yarn: 1.22.0 - C:\Program Files\nodejs\yarn.CMD
npm: 6.13.4 - C:\Program Files\nodejs\npm.CMD
npmPackages:
istanbul-instrumenter-loader: 1.2.0 => 1.2.0
istanbul-lib-coverage: 1.2.1 => 1.2.1
nyc: ^15.0.1 => 15.0.1
source-map-support: ^0.5.16 => 0.5.16
ts-node: ^7.0.1 => 7.0.1
typescript: ~3.7.4 => 3.7.5
For Cypress coverage I use above dependencies as well as @cypress/code-coverage:^3.0.2
.
I’m using jest: 24.1.0
to produce unit test coverage.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 4
- Comments: 22 (1 by maintainers)
Commits related to this issue
- Add Visual Coverage Test + Combined Code Coverage for Coveralls (#275) Use storybook test-runner and `@storybook/addon-coverage` to get visual coverage report. The setup requires spinning up storyboo... — committed to yext/search-ui-react by yen-tt 2 years ago
- Add Visual Coverage Test + Combined Code Coverage for Coveralls (#275) Use storybook test-runner and `@storybook/addon-coverage` to get visual coverage report. The setup requires spinning up storyboo... — committed to yext/search-ui-react by yen-tt 2 years ago
I’m experiencing this same problem.
My coverage results look like so:
cypress/coverage-final.json
:jest/coverage-final.json
:The expected outcome would be
path/to/file.ts
having 100% coverage, since the gaps on one test are covered by the other test. However, instead, what you see is the gaps being reported:combined/coverage-final.json
:The “coverage” merge is really just concatenated instead of merging.
It appears that:
line 1 characters 1-10 uncovered
+line 1 characters 1-5 covered
+line 1 characters 5-10 covered
=line 1 characters 1-10 uncovered
(wrong) +line 1 characters 1-10 covered
(right).The uncovered characters are still shown as uncovered even though their character-span is covered by a different part of the map. Just guessing.
Here is a nice visual LCOV report for the above JSON:
Cypress (fully covered except for 1 branch):
Jest (specifically covering the above branch):
Combined report (expected to be fully covered):
Any advice to unblock this? I don’t want to have to write a JSON merger manually. 😓
Maybe this is just a weird issue with TypeScript.
Below, how are imports and exports not covered? The
export
statement isn’t covered, but the inside of the function somehow is. The inside references an import, but the import isn’t covered. Nonsensical. 😵I have the same issue just by merging multiple reports from various cypress runs in parallel. The merge is just taking the last report base on the order:
coverage-final-A.json
coverage-final-B.json
In this case, only the result of
coverage-final-A.json
will be taken, so without coverage. If I renamecoverage-final-A.json
tocoverage-final-Z.json
, then the order is different and the result ofcoverage-final-B.json
will be taken.Any ideas? Should rewrite a merge myself?
Hi all, Maybe it will help someone: I had the same problem (merging of cypress and jest coverage wasn’t right) and I got that fixed after adding coverageProvider into jest config:
coverageProvider: 'v8',
@D0rmouse did you finally find a good configuration to combine multiple coverage reports?
I have the same type of stack with Nx + Cypress + Storybook + Jest, generating reports separately works fine but when I want to merge, it is not good.
It makes sense because each tool are using different versions for each generation.
I’m encountering the same problem. When I try to merge two coverage-final.json files, the output is erroneous.
I have posted a Stack Overflow question with the details here .
In my case, I fixed this issue by using the babel coverage provider in my Jest config
coverageProvider: 'babel',
. In my babel config I added a enviroment variable to check if the Code should be instrumented or not.To run this in instrument mode, add a script like this to your package.json
"dev:test": "set INSTRUMENT_CODE=1 && next dev",
This finally provides accurate coverage results for me.
Same issue. Any updates. Screenshots below:
Cypress Coverage - correct:
Jest Coverage - correct:
Combined Coverage - wrong:
I’m using nyc merge and nyc report just like the original reporter of this issue. The final JSON just doesn’t appear correct.
@mmisty I tried your suggestion and it didn’t work. I end up with this instead: