jest: [Bug]: Generating a coverage report with --runInBand, collectCoverageFrom, and a transformer can mask a failing exit code

Version

29.0.2

Steps to reproduce

  1. Create a new project that uses a jest transformer (e.g. ts-jest).

  2. Specify a collectCoverageFrom in jest.config.js in a way that will invoke the transformer when collecting coverage

jest.config.js

module.exports = {
  collectCoverageFrom: ["src/**/*.ts"],
  preset: 'ts-jest',
};
  1. Create a foo.test.js file that will fail to run (e.g. including a syntax error). Note, this does NOT need to pass through the transformer. e.g.:

foo.test.js

syntaxError!;
  1. Create a source file with a name that matches the collectCoverageFrom pattern and will pass through the jest transformer (e.g. bar.ts), and write something that will cause an error, e.g. a syntax error, in that file.

bar.ts

anotherSyntaxErrror!;
  1. Run npx jest --runInBand --coverage

  2. Run echo $? to see the exit code, which will be 0

Expected behavior

I expect:

  1. the coverage report to generate and
  2. for the exit code to be 1.

Actual behavior

The tests fail (due to failure to run) but the coverage report generation silently errors and the exit code for the entire process is, incorrectly,0.

(This, for instance, means that CI marks tests as succeeding).

Additional context

Running npx jest --coverage without --runInBand does not cause this bug and instead renders output such as:

Running coverage on untested files...Failed to collect coverage from /Users/slifty/Maestral/Code/personal/jesttest/src/foo.ts
ERROR: Jest worker encountered 3 child process exceptions, exceeding retry limit
STACK: Error: Jest worker encountered 3 child process exceptions, exceeding retry limit
    at ChildProcessWorker.initialize (/Users/slifty/Maestral/Code/personal/jesttest/node_modules/jest-worker/build/workers/ChildProcessWorker.js:211:21)
    at ChildProcessWorker._onExit (/Users/slifty/Maestral/Code/personal/jesttest/node_modules/jest-worker/build/workers/ChildProcessWorker.js:396:12)
    at ChildProcess.emit (node:events:513:28)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|-------------------
All files |       0 |        0 |       0 |       0 |
----------|---------|----------|---------|---------|-------------------
Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        4 s
Ran all test suites.

Running npx jest --runInBand --coverage only renders:

Running coverage on untested files...

and then the process exits with code 0.

Some other interesting “alternative outcomes”:

  • If collectCoverageFrom is not specified then coverage is generated as expected and the process returns correct exit codes.
  • If the tests are able to run then then coverage is STILL not generated and the exit code is always 1 regardless of test outcomes.

Environment

System:
    OS: macOS 12.4
    CPU: (8) x64 Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz
  Binaries:
    Node: 18.9.0 - ~/.nvm/versions/node/v18.9.0/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 8.19.1 - ~/.nvm/versions/node/v18.9.0/bin/npm
  npmPackages:
    jest: ^29.0.2 => 29.0.2

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 12
  • Comments: 33 (2 by maintainers)

Commits related to this issue

Most upvoted comments

As far as i know, this is still an issue

I can confirm that the bug is still there with the latest version of jest and ts-jest as of May 2023. Also I found that --maxWorkers=2 (without runInBand) fixes the problem.

I have a test case covering this issue: https://github.com/handy-common-utils/dev-dependencies/blob/08fc16a45db1e22882f084f14c3be4acaca1e956/jest/test/fs-utils.spec.ts#LL51C7-L51C67

And the test case would fail in GitHub actions if I remove --maxWorkers=2: https://github.com/handy-common-utils/dev-dependencies/blob/08fc16a45db1e22882f084f14c3be4acaca1e956/jest/test/fixtures/fs-utils/package.json#L7

I suspect that GitHub gives the actions 2 virtual CPU cores, and that triggers the problem if you don’t tell Jest to use 2 workers.

  1. Create a new project that uses a jest transformer (e.g. ts-jest).

Is the babel-jest transformer causing this problem as well?

I am asking this because babel-jest is the only transformer in the Jest repo. If only some other transformers are causing this issue, that could mean this bug is on their side.

Having looked at no code, here’s my hopefully-not-red-herring pet theory in lieu of triage.

I noticed similar behavior setting --maxWorkers=1.

I believe the intention (feel like this was implied in the doc somewhere) is for there to be the main “thread” plus one or more workers. I assume the trouble comes in when the main “thread” is the only one. It thinks it’s a worker, and ends for whatever reason with no one left to do the clean-up, reporting, whatever.

If my imagination is in the ballpark, a solution might be to enforce the correct minimum worker count (guessing 2), and/or ensure that when the main “thread” is working as a solitary worker that it’s resilient enough to pick back up and do main thread stuff whether its worker work was successful or not (think try/catch/finally sort of thing).

I’m also hitting this now. (I know “+1 isn’t helpful”, but stalebot is watching the clock! 😉 )