prisma: Memory leaks in Jest when running tests serially with `nApi` enabled

I had a look at this today. I think this is happening because we keep a collection of engines around here https://github.com/prisma/prisma/blob/f395abad8eb85739a1df35071caa9e5050993696/packages/engine-core/src/library/LibraryEngine.ts#L44

And each time we create a new PrismaClient we create a new engine. This leads to a duplication of the whole engine which creates a lot of memory. I think we need to change the way we create the Library engine to only use one engine.

Any news? This is quite a critical bug for anyone that uses Prisma in prod with a CI pipeline

I can confirm my company’s Jest CI tests started leaking like crazy after upgrading to Prisma 3.x. This was the only changed variable so there’s definitely something happening. I am trying to reproduce with a simpler setup.

@Ustice thanks for looking into it a bit. The prisma client team has been doing some deep diving into this. We have found a few things. Using the engine or mocking it out doesn’t affect the heap size that much. We have tried a few things around the loading and removing of the engine and it made no difference. I also created another test sample and only loaded the engine without the typescript side and then there was no heap memory increase.

So we looking in a few places to solve this. We have some ideas and doing a fair amount of experimenting with different ideas to fix this.

@garrensmith not sure why that would be an issue given that jest sandboxes modules between test files.

@aqrln Those observations are somewhat correct in the sense that running --logHeapUsage without exposing node’s garbage collector won’t surface any abnormal usage patterns for very small sample sizes (n=10 in the case of your fork) - because jest won’t be able to free up unused memory as soon as possible (https://github.com/facebook/jest/blob/9edcaaffd1cdf418feca6d983cdfd364b73bf62c/packages/jest-runner/src/runTest.ts#L320).

As a follow up from my previous comment:

I’m still unsure of what the ~100mb of leaked memory consists of, for the larger project. I’ll try to set up a more realistic example to see if I can get a definitive answer.

Looks like the prisma client’s in-memory size scales with the schema, which would explain why the minimal reproduction didn’t originally exhibit as severe a increase in heap size.

I’ve updated the original repo’s README with some additional metrics that showcase the issue in more detail. Pasting the important ones here:

Comparison against binary engine for small schema

Comparison against the modified library engine with logging disabled

The setup was as minimal as possible:

• node v14
• vanilla js
• prisma.$connect to instantiate the engine, no other operations

I’ve also checked the modified engine into version control, in case anyone wants to give GH actions a spin: https://github.com/driimus/prisma-leaks/tree/bench/libengine-without-logging

We are experiencing very high memory usage too, actually 1.4gb after just a single test suite (around 60 integration tests).

I like the idea of $dispose. Lets see what others have to say here.

@janpio

Is there a feature request issue for this, or the overall “Limitations” section?

Not as of yet. I’m waiting on #4703 to explore test parallelization in more detail, since re-applying migrations plays a big part in using separate database schemas per test suite.

In your understanding our (node-api library) logging implementation is leaking memory?

It’s a fairly big assumption on my part, but yes, I think the logging component causes leaks in jest test suites.

My understanding of Node-API and the query engine is very limited, but this line in particular was the main reason I started looking at how jest reacts to getting rid of the logging callback. This was entirely based on the documentation for napi_unref_threadsafe_function, which states:

the event loop running on the main thread may exit before func is destroyed

As that seemed like something that the weak-napi package used by jest might detect.

@pimeys I’ve played around a fair bit with instantiating a QueryEngine when I was testing changes to my stubbed out version in that repo:

• instantiating inside test closures (with a single test inside the suite)
• instantiating at the top-level of the entire test suite

None of those have made a difference as far as the --detectLeaks flag is concerned.

More importantly:

• the project that I’m trying to migrate makes use of a singleton Prisma client (without any custom settings)
• the minimal reproduction repo instantiates a single client per suite, and this step runs a single test.

I’ve made two more changes to the minimal reproduction repo, one for importing a client, one for removing explicit disconnects. Hopefully the additional scenarios provide more insight.

Please open a new issue and provide additional information. Your leak is 99.9% not the same we fixed via this issue a year ago. Thanks.

We are experiencing memory leaks with:

• Node 18
• Prisma 4.11.0

When we inspect our heap its littered with incrementally expanding Prisma strings. This crashes our CI pipeline.

I ran some tests on my machines using the reproduction repo by @driimus: https://github.com/aqrln/prisma-leaks/blob/main/stats.md.

As @janpio noted in https://github.com/prisma/prisma/issues/8989#issuecomment-910711794, there doesn’t seem to be visible correlation between memory usage and N-API in this specific test: both library and binary engines produce similar numbers. It also looks like the memory might have been leaking in tests even before 3.0.0, it just wasn’t as noticeable. However, it does look like the memory usage has become worse in the recent versions on Linux, regardless of whether N-API is used or not, and it didn’t just happen once around 3.0.0: the numbers are even higher on 3.6.0.

In addition to that, Node.js version also affects numbers a lot. The drastic increase in memory usage with Prisma 3.6.0 on Linux seems to start happening on Node.js 16 and later (https://github.com/aqrln/prisma-leaks/blob/main/stats-by-version.md), which is consistent with the fact that the issue only became visible on our CI once we added Node.js 16 to the matrix: https://github.com/prisma/prisma/issues/10183.

The next steps would be to actually profile JavaScript heap allocations and memory usage and see what’s happening there.

We can reproduce the memory leak in CI (GitHub Actions) with Node.js v16 here https://github.com/prisma/prisma/pull/10020

We are having memory leaks in our pipeline too

This is making it impossible to run our test suite in our CI pipeline. Please keep this issue open.

I haven’t tried any real v4 tests yet, but here’s a test run for v4.1, which pertains to this specific issue https://github.com/driimus/prisma-leaks/actions/runs/2699611305 (take your pick of node version, schema size, test file count).

Unless you have a significant amount of test files, the problem might lie somewhere else in your tests, as the loss now seems to be insignificant at ~2-3MB per file.

@ethancadoo are you running jest with --expose-gc flag by any chance? If so, there is a separate issue on v8 side, try running the tests without this flag. If it you are not using it (or removing it does not help), please, provide a reproduction — cases, reported here should be fixed as of 4.1.0.

I’m running isolated tests in prisma using vitest and https://www.npmjs.com/package/@chax-at/transactional-prisma-testing It mostly works

Thanks guys for your hard work on this issue

Any updates? I currently suffer from memory leaks on my server written in Nest.js when I run tests. GitHub Actions is crashing.

@garrensmith Thanks for pointing out where the problem was. This issue was blocking our adoption of Prisma. I created a PR for a fix. It refactors the way that the beforeExit is handled so that there aren’t any undead references to the engines., allowing garbage collection.

There are failing tests in src/__tests__/MigrateDiff.test.ts, but it looks like that is a bugged test? I’m not familiar enough with the overall codebase to know.

@driimus I’m pretty sure that the real leak is from here. There are references to the engines in process event handlers. While jest isolates modules (mostly), references to process can and do leak.

On my app it happens in Jest in NODE_ENV=test

1. The first commit only helped get past --detectLeaks errors when instantiating a query engine. The second one got the tests in https://github.com/driimus/prisma-leaks to pass without an increase in heap usage.

2. I haven’t gotten as far as monitoring nApi behavior in a live environment. I’d consider minor leaks for long-running clients to be a separate issue, unless there’s a correlation between them and the amount of data being transferred in and out. As for explicitly calling $disconnect: if that’s the intended method of marking the engine for garbage collection, I could see that being a problem for Jest users (see 4).

3. Pretty much.

4. As far as I know, Jest resets imported modules between test files (there’s also an even more aggressive sandboxing mechanism available through resetModules - though I haven’t checked whether that could impact the issue’s severity). Even if an instance of the Prisma Client is imported, there will be a separate copy per test file. I’m still unsure of what the ~100mb of leaked memory consists of, for the larger project. I’ll try to set up a more realistic example to see if I can get a definitive answer.

@pimeys Thanks for mention, I will dig into this problem in this week.