jest: Jest memory problems in node environment

@rickhanlonii, I’m also experiencing this bug, and I do suspect it’s a bug with jest’s module registry.

Jest does not cache modules as node’s require() does and probably does not release properly require()'d modules between test suites.

Created a repo to illustrate this issue.

@SimenB @cpojer @aaronabramov the source of the issue comes from fact that for every test run new modules cache is created but it’s used for all but native node.js modules.

On example of graceful-fs, following happens when more than one test is run within same process (or worker) as handled by Jest:

1. (first test run) graceful-fs is freshly required and decorates native fs functions directly on fs object (note: through closures graceful-fs is permanently attached to fs and therefore cannot be freed)
2. (second test run) graceful-fs is freshly required (again) within same process, and decorates (already decorated!) fs functions directly on fs object. Same way, it’s permanently attached to fs so not freed.

… scenario repeats with every following test run.

So after a few test runs we deal with a situation, where after invoking single specific fs function, multiple graceful-fs functions are run recursively, and that I suppose is responsible for fastly growing memory leak.

Similarly popular Bluebird.promisifyAll(require('fs')) attaches to fs and introduces a leak. Still in that case it’s not recursive so not that harmful.

I’ve noticed this when trying to incorporate long stack trace solution (based on async_hooks) into Jest, and observed how quickly it was blown with OOM, cause was that initializations were made before each test run, when there should be one initialization per process

When looking at Jest design, probably best solution would be to introduce an option in which we can list package names which should be handled by nodejs native modules loader (and graceful-fs could be there by default), so they’re loaded once across all test runs within same process. As far as I see it’s not possible now, and that makes addressing this difficult.

Other solutions could be:

• Provide an option to rely directly on require as provided by Node.js. It won’t give desired isolation, but at least it’ll reflect public Node.js API, which is not respected with current approach.
• Provide an option to run each test in its designated worker . It’ll be slower, but it’s probably the only way to provide perfect side-effects free isolation.

This is something that only Jest could solve in the long run, relying on external authors to not do the bad thing (or even be aware of the issue) is just keeping a door open for this to happen again anyways in the future.

Is there something we could help on about this to make the matter advance?

@lev-kazakov @mrrinot Fixed the issue on our project by replacing graceful-fs with fs when running the tests. (We change the file in the node_modules directory). It fixed the issue completely. We use this as a replacement:

const fs = require('fs');

module.exports = Object.assign(fs, { gracefulify: () => {} });

We put this inside node_modules/graceful-fs/graceful-fs.js when the tests start and we restore the “real” version when they are done.

It seems that graceful is only useful when you are on Windows. Since we don’t use Windows at all, we don’t need it.

@SimenB, I’ve sanitised a test case with no external dependencies: https://github.com/lev-kazakov/jest-leak/tree/master

If you run yarn jasmine, you’ll see the memory is stabile around 7 Mb. If you run node --expose-gc ./node_modules/.bin/jest --runInBand --logHeapUsage, you’ll see memory leaking.

Memory leaks when you require code that overrides one of node’s internal modules functions, while keeping a reference to the original function, like here.

Libraries that do this kind of hacks:

• graceful-fs
• agent-base (puppeteer dep)

Would you like me to open a new ticket?

Hi guys, is there any update on this?

Maybe coming with node 12 LTS and the worker modules? I have been bit hard by this issue yesterday and have been trying to find a solution since and still no luck 😢

I know that graceful-fs has been patched in 4.1.12 but even by forcing the resolved version through yarn to 4.1.15 I still face the issue so I suppose other packages in my node_modules are doing this.

Sadly I don’t know of a good method to pinpoint those unless I spend a week on it and I can’t afford that right now.

Is there any update or plan for this in a near future?

@SimenB, @mrrinot ,

I think I’ve sanitised a test case that proves graceful-fs is not the issue, but rather it’s graceful-fs when required via jest’s module registry.

The following repo illustrates the issue: https://github.com/lev-kazakov/jest-leak/tree/master

If you run yarn jasmine, you’ll see the memory is stabile around 7 Mb. If you run node --expose-gc ./node_modules/.bin/jest --runInBand --logHeapUsage, you’ll see memory leaking.

Note that I’m using the import-fresh module to bypass node’s cache when running with jasmine.

Here is a quick repro if you want: https://github.com/mrrinot/fs-extra-jest-repro After a more thorough test, it seems it is graceful-fs instead. Issues that seem related:

• https://github.com/isaacs/node-graceful-fs/issues/102
• https://github.com/facebook/jest/issues/5239

It doesn’t seem that graceful-fs as of version 4.1.15 has actually fixed the memory leak. I’ve forced npm to use graceful-fs@4.1.15 throughout, but doing that my tests still leak memory like crazy and eventually crash. However, when I replace graceful-fs/graceful-fs.js with the replacement specified by @Telokis then my tests stop leaking memory and finish with no problem:

const fs = require('fs');

module.exports = Object.assign(fs, { gracefulify: () => {} });

I see a comment in graceful-fs.js source code where it seems like they’ve tried to stop leaking memory but clearly their fix is not working:

// Only patch fs once, otherwise we'll run into a memory leak if
// graceful-fs is loaded multiple times, such as in test environments that
// reset the loaded modules between tests.

@lev-kazakov Thanks. I removed the --verbose flag but the memory leak is still present…

Guys, check if you have properly setup paths where JEST is scanning files for coverage. I’ve just discovered it goes to “dist” folder and this is causing memory looping - after excluding it everything works like a charm again.

@seyfer you can run your test suite against https://github.com/facebook/jest/pull/8331. run with --freezeCoreModules and --verbose and it will smoke out the leaking modules. in order to run it against https://github.com/facebook/jest/pull/8331 you can either checkout the branch then build and link jest, or just hack your local node_modules/jest-runtime/build/index.js file with those changes: https://github.com/facebook/jest/pull/8331/files#diff-5b473ffb7e91f9fc070a6f6025790ef0 here’s an outdated but working patch for this matter: https://gist.github.com/sibelius/f98e62dd9346ddc97f07fe3814dc1b6e.

I’d vote for implementing this first:

Provide an option to rely directly on require as provided by Node.js. It won’t give desired isolation, but at least it’ll reflect public Node.js API, which is not respected with current approach.

And then this:

Provide an option to run each test in its designated worker . It’ll be slower, but it’s probably the only way to provide perfect side-effects free isolation.

@lev-kazakov I didn’t do anything except reporting here. @mrrinot did the work alone in our office’s cave.

yeah. anyhow i’ve created a patcher inspired by @Telokis comment above. brute hack, but works. https://github.com/lev-kazakov/jest-leak-fixer.

Jest has built in --logHeapUsage, fwiw

Ok, turns out the problem comes from us importing fs-extra in one of our model. We use the latest version (6.0.1).

Just commenting the import fixes the memory going up between test suites. I don’t really know if it’s a problem with fs-extra or jest.

Quick update:

I “solved” (more like put it under the rug) my issue by splitting the work done by jest beforehand.

find . -iname '*spec*' | grep -v node_modules | xargs -n 5 jest --no-cache --globalSetup ./test/globalSetup.js --forceExit

Of course this sucks because I can’t take advantage of concurrent testing, and my test suite takes forever, but at least it is able to run till the end.

I’m sorry to up this again, but do we have any updates on the matter?

it’s graceful-fs: https://github.com/isaacs/node-graceful-fs/issues/102. @SimenB, fyi, jest is also dependent on graceful-fs.

I’m working on a reduced case of the problem, will get back to you when/if needed