vitest: Unexpected Istanbul code coverage report

Describe the bug

In my experience Vitest produces an unexpected code coverage report with Istanbul. In general, when I have only unit tests in my applicaiton, Vitest+ Istanbul outputs a perfect code coverage report. However, if I do anything more complex, the code coverage report is incorrect and unexplainable.

For example, I a have a Fastify application. The moment I do testing with Fastify (i.e. not strict unit tests), the code coverage go haywire. I am comparing Vitest + Istanbul to Node TAP (NYC + Istanbul).

Coverage report with Node TAP:

-----------------|---------|----------|---------|---------|-------------------
File             | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------------|---------|----------|---------|---------|-------------------
All files        |    92.3 |       75 |   88.88 |   91.66 |                   
 src             |     100 |      100 |     100 |     100 |                   
  app.ts         |     100 |      100 |     100 |     100 |                   
 src/lib         |    87.5 |       75 |     100 |    87.5 |                   
  foo-bar.ts     |     100 |      100 |     100 |     100 |                   
  hello-world.ts |      75 |       50 |     100 |      75 | 5                 
 src/routes      |     100 |      100 |     100 |     100 |                   
  root.ts        |     100 |      100 |     100 |     100 |                   
 src/routes/foo  |      80 |      100 |   66.66 |      75 |                   
  index.ts       |      80 |      100 |   66.66 |      75 | 6                 
-----------------|---------|----------|---------|---------|-------------------

Coverage report with Vitest + Istanbul:

-----------------|---------|----------|---------|---------|-------------------
File             | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------------|---------|----------|---------|---------|-------------------
All files        |   13.63 |        0 |    12.5 |   13.63 |                   
 src             |   27.27 |      100 |      50 |   27.27 |                   
  app.ts         |     100 |      100 |     100 |     100 |                   
  server.ts      |       0 |      100 |       0 |       0 | 2-12              
 src/lib         |       0 |        0 |       0 |       0 |                   
  foo-bar.ts     |       0 |        0 |       0 |       0 | 2-5               
  hello-world.ts |       0 |        0 |       0 |       0 | 2-5               
 src/routes      |       0 |      100 |       0 |       0 |                   
  root.ts        |       0 |      100 |       0 |       0 | 3-5               
 src/routes/foo  |       0 |      100 |       0 |       0 |                   
  index.ts       |       0 |      100 |       0 |       0 | 3-4               
-----------------|---------|----------|---------|---------|-------------------

I am not sure what would be causing such a dramatic difference in coverage report. Any ideas?

Reproduction

I setup a GitHub repo that you can clone to see the above issue: https://github.com/kylerush/vitest-v-tap

System Info

System:
    OS: macOS 12.5.1
    CPU: (10) arm64 Apple M1 Pro
    Memory: 321.05 MB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.7.0 - ~/.nvm/versions/node/v18.7.0/bin/node
    npm: 8.15.0 - ~/.nvm/versions/node/v18.7.0/bin/npm
  Browsers:
    Chrome: 99.0.4844.51
    Firefox: 103.0.2
    Safari: 15.6.1
  npmPackages:
    vite: ^3.1.0 => 3.1.0 
    vitest: ^0.23.1 => 0.23.1

Used Package Manager

npm

Validations

Follow our Code of Conduct
Read the Contributing Guidelines.
Read the docs.
Check that there isn’t already an issue that reports the same bug to avoid creating a duplicate.
Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
The provided reproduction is a minimal reproducible example of the bug.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 23 (14 by maintainers)

Most upvoted comments

Yes, the coverage is correct with Jest. The uncovered lines in Jest’s report matches TAP’s report. The overall % discrepancy I suspect has something to do with how TAP and Jest handle TypeScript, but not certain.

Jest:

-----------------|---------|----------|---------|---------|-------------------
File             | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------------|---------|----------|---------|---------|-------------------
All files        |   85.71 |       75 |   85.71 |   85.71 |                   
 src             |     100 |      100 |     100 |     100 |                   
  app.ts         |     100 |      100 |     100 |     100 |                   
 src/lib         |   83.33 |       75 |     100 |   83.33 |                   
  foo-bar.ts     |     100 |      100 |     100 |     100 |                   
  hello-world.ts |   66.66 |       50 |     100 |   66.66 | 5                 
 src/routes      |     100 |      100 |     100 |     100 |                   
  root.ts        |     100 |      100 |     100 |     100 |                   
 src/routes/foo  |      50 |      100 |      50 |      50 |                   
  index.ts       |      50 |      100 |      50 |      50 | 6                 
-----------------|---------|----------|---------|---------|-------------------

TAP:

-----------------|---------|----------|---------|---------|-------------------
File             | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------------|---------|----------|---------|---------|-------------------
All files        |    92.3 |       75 |   88.88 |   91.66 |                   
 src             |     100 |      100 |     100 |     100 |                   
  app.ts         |     100 |      100 |     100 |     100 |                   
 src/lib         |    87.5 |       75 |     100 |    87.5 |                   
  foo-bar.ts     |     100 |      100 |     100 |     100 |                   
  hello-world.ts |      75 |       50 |     100 |      75 | 5                 
 src/routes      |     100 |      100 |     100 |     100 |                   
  root.ts        |     100 |      100 |     100 |     100 |                   
 src/routes/foo  |      80 |      100 |   66.66 |      75 |                   
  index.ts       |      80 |      100 |   66.66 |      75 | 6                 
-----------------|---------|----------|---------|---------|-------------------

Vitest:

-----------------|---------|----------|---------|---------|-------------------
File             | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------------|---------|----------|---------|---------|-------------------
All files        |   13.63 |        0 |    12.5 |   13.63 |                   
 src             |   27.27 |      100 |      50 |   27.27 |                   
  app.ts         |     100 |      100 |     100 |     100 |                   
  server.ts      |       0 |      100 |       0 |       0 | 2-12              
 src/lib         |       0 |        0 |       0 |       0 |                   
  foo-bar.ts     |       0 |        0 |       0 |       0 | 2-5               
  hello-world.ts |       0 |        0 |       0 |       0 | 2-5               
 src/routes      |       0 |      100 |       0 |       0 |                   
  root.ts        |       0 |      100 |       0 |       0 | 3-5               
 src/routes/foo  |       0 |      100 |       0 |       0 |                   
  index.ts       |       0 |      100 |       0 |       0 | 3-4               
-----------------|---------|----------|---------|---------|-------------------

kylerush on Sep 12, 2022

When @fastify/autoload is loading the routes, these are not loaded through vitest’s Vite plugins. nyc has something called istanbul-lib-hook which might be related to this but as it does not have documentation it’s a bit tricky to use.

Is the coverage correct when using Jest?

AriPerkkio on Sep 12, 2022