jest: [Bug]: Memory Leak with --runInBand

Version

27.4.7, ^28.0.3

Steps to reproduce

https://github.com/dpchamps/jest-oom-repro

To summarize the results of the repro:

--runInBand seems to prevent GC between specifications. While the native contexts repro might seem to be specifically related to jest-environment-jsdom, the v8 regex literal bug warrants an investigation into the test runner, as there could be some shared context that isn’t properly getting cleaned up.

It’s worth mentioning that the reason to suspect there is something shared at the jest test runner level (outside of say, jest-environment-jsdom) has to do with what surfaces the regex literal oom error. You can see a more targeted reproduction of this issue here: https://github.com/dpchamps/node-regex-oom. The issue seems to only be triggered by regex literals that have been used more than once. I wouldn’t expect to see specifications that create new regex literals failing with this error, but when executed with --runInBand there appears to be a significant performance hit as more specifications are run, which will lead to the eventual Regex OOM error.

This happens with both jsdom and node environments.

Expected behavior

I expect to see isolated environments between specification runs to be properly garbage collected when run in band

Actual behavior

As outlined in the repro charts here, there is a memory leak.

Additional context

It might seem silly to demonstrate this repro against 1000 specifications, but we actually run into this problem at Walmart with a few suites that are quite large (not 1000 specs large, but still, we’re feeling this pain).

These charts are included in the MVR, but will dupe them here for quick access

jest-memory-leak-run-in-band number-of-native-contexts

Environment

System:
    OS: macOS 12.0
    CPU: (10) arm64 Apple M1 Pro
  Binaries:
    Node: 16.15.0 - ~/.nvm/versions/node/v16.15.0/bin/node
    Yarn: 1.22.17 - ~/.nvm/versions/node/v16.6.2/bin/yarn
    npm: 8.5.5 - ~/.nvm/versions/node/v16.15.0/bin/npm
  npmPackages:
    jest: ^28.0.3 => 28.0.3

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 12
Comments: 20

Most upvoted comments

Thanks @jmoore240, I have imlemented your workaround and it works like a charm. I also added some more options to my jest.config.js

{
  cache: false,
  maxWorkers: "50%",
  maxConcurrency: 3,
  transform: {
    "^.+\\.(t|j)s$": "@swc/jest"
  },
}

command: node --no-compilation-cache --expose-gc ./node_modules/jest/bin/jest.js

With this setup I can reduce the memory usage down from 8 GB to just 300 MB per test suite. Btw I used @swc/core to further speed up the test runner.

I also have stumbled upon other solution such as https://github.com/goloveychuk/fastest-jest-runner which doesn’t involve calling jest from node with custom arguments, but I don’t like adding new dependencies just for this issue.

  System:
    OS: Linux 5.15 Ubuntu 20.04.4 LTS (Focal Fossa)
    CPU: (12) x64 Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz
  Binaries:
    Node: 18.12.1 - ~/.nvm/versions/node/v18.12.1/bin/node
    Yarn: 1.22.19 - ~/.local/share/pnpm/yarn
    npm: 9.2.0 - ~/.nvm/versions/node/v18.12.1/bin/npm
  npmPackages:
    jest: ^29.3.1 => 29.3.1

fahminlb33 on Jan 25, 2023

For those experiencing this issue, the following node and jest flags help keep the heap down by running the gc after tests. Memory leak is still there, but this is a work around that has been good enough for us in the meantime.

NodeJS Flags: --expose-gc --no-compilation-cache --max-old-space-size=700 Jest Flags: --no-cache --logHeapUsage

Hope this helps!

jmoore240 on Jan 11, 2023

It would be great to get acknowledgement from jest dev team so we can submit a patch.

tchon on Sep 13, 2022