Our team ran into a similar issue with our NestJS application. The memory leak is very slow (about 800MB in 48 hours). Approximate server load information:

• 2.5K inserts per second
• 22K selects per second
• 2.2K deletions per second
• 280 updates per second

We tried several options to localize the memory leak. Here is the result we got. When running the application with engineType "library" nodeJS process keep on consume memory. When using engineType "binary" nodeJS application uses memory correctly but the prisma-query-engine process continues to consume memory. When comparing 2 different launch methods, engineType "library" leaks about twice as fast compared to engineType "binary". We are using prisma version 5.2.0, node v16.18.1.

OS info: Linux 5.4.179-1.el8.elrepo.x86_64 #1 SMP Thu Feb 10 10:14:12 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

Memory usage:

Prisma info:

Memory usage chart (engineType "binary")

I can confirm there is some memory leak in the Prisma query engine. It affects all our apps which use Prisma and make a lot of DB calls regardless of whether they are read-only apps or the ones fetching external data and storing it to the DB.

I thought Prisma 5 might solve this problem but after I updated the dependencies today, I can see the issue is still there.

The memory used by our apps slowly grows day-by-day until they eventually run out of memory and their containers are restarted.

Below I show some examples what this problem looks like in both library and binary mode. In these cases, I simply hit some of our API endpoints that make really heavy DB calls so I can reproduce the issue more quickly.

Library

When we use library engine type, the issue is only visible as the difference between rss memory usage.

You can clearly see there’s no memory leak in our app or in Prisma client because the heap size remains stable.

Before Heavy Load

{
  "rss": 336392192,
  "heapTotal": 33280000,
  "heapUsed": 27778056,
  "external": 17858813,
  "arrayBuffers": 51654
}

After Heavy Load

{
  "rss": 571432960,
  "heapTotal": 30396416,
  "heapUsed": 26941400,
  "external": 17875669,
  "arrayBuffers": 39294
}

Binary

When we use binary engine type, it is clearly visible that the query engine uses a lot of memory.

Used memory is shown as percentage below. But it’s easy to calculate how much memory those processes use since the machine where the app runs has 1GB RAM.

I only made a single set of heavy DB calls. But I can repeat this over and over again with the very same calls and I always end up with more and more memory used by the query engine.

After Start

After I start the application, the Node.js memory usage looks like this:

{
  "rss": 154386432,
  "heapTotal": 30298112,
  "heapUsed": 26703576,
  "external": 18142813,
  "arrayBuffers": 56092
}

…and relevant system processes like this:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
apps        76  7.9 15.3 21784396 161136 ?     Sl   09:03   0:05 /layers/heroku_nodejs-engine/nodejs/bin/node --inspect=127.0.0.1:9221 --heapsnapshot-signal=SIGUSR2 build/server.js
apps        88  0.6  4.7 795948 49700 ?        Sl   09:03   0:00 /workspace/node_modules/.prisma/client/query-engine-debian-openssl-1.0.x --enable-raw-queries --enable-metrics --enable-open-telemetry --port 0 --engine-protocol json

After Heat Up Period

I let the application run for a while but there’s nothing going on except for regular health checks (which also connect to the DB).

{
  "rss": 158240768,
  "heapTotal": 31870976,
  "heapUsed": 28238904,
  "external": 18340681,
  "arrayBuffers": 65628
}

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
apps        76  4.1 14.7 21783536 154532 ?     Sl   09:03   0:40 /layers/heroku_nodejs-engine/nodejs/bin/node --inspect=127.0.0.1:9221 --heapsnapshot-signal=SIGUSR2 build/server.js
apps        88  2.4 14.5 798000 152492 ?       Sl   09:03   0:22 /workspace/node_modules/.prisma/client/query-engine-debian-openssl-1.0.x --enable-raw-queries --enable-metrics --enable-open-telemetry --port 0 --engine-protocol json

After Heavy Load

{
  "rss": 207368192,
  "heapTotal": 80367616,
  "heapUsed": 69357296,
  "external": 18272118,
  "arrayBuffers": 63632
}

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
apps        76  5.3 19.3 21823728 202508 ?     Sl   09:03   1:03 /layers/heroku_nodejs-engine/nodejs/bin/node --inspect=127.0.0.1:9221 --heapsnapshot-signal=SIGUSR2 build/server.js
apps        88  3.2 29.6 863536 310560 ?       Sl   09:03   0:37 /workspace/node_modules/.prisma/client/query-engine-debian-openssl-1.0.x --enable-raw-queries --enable-metrics --enable-open-telemetry --port 0 --engine-protocol json

After Cool Down Period

When I wait for a while after the heavy DB calls, I can see that GC kicks in and cleans up the heap. However, the memory used by the Prisma query engine doesn’t go down.

{
  "rss": 158236672,
  "heapTotal": 32133120,
  "heapUsed": 27977328,
  "external": 18275459,
  "arrayBuffers": 52850
}

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
apps        76  5.0 14.7 21783792 154528 ?     Sl   09:03   1:05 /layers/heroku_nodejs-engine/nodejs/bin/node --inspect=127.0.0.1:9221 --heapsnapshot-signal=SIGUSR2 build/server.js
apps        88  2.9 29.6 863536 310560 ?       Sl   09:03   0:37 /workspace/node_modules/.prisma/client/query-engine-debian-openssl-1.0.x --enable-raw-queries --enable-metrics --enable-open-telemetry --port 0 --engine-protocol json

Can confirm: Upgrading to Node v20 and Prisma v5.5.2 fixed the leak for me. Previously, memory usage would increase to 2GB within a few hours. Now, it remains at approximately 600MB over the same time span.

Same here, my issues have been resolved since the node upgrade. 5.5.2 & Node 18.18.2 Thanks to whoever fixed it, accidentally or on purpose!

In case folks missed it, this looks like its related to using Prisma on Node 16/18. Updating to Nodejs v20 fixed the memory leak for my team.

I’m sorry I know this is not a particularly helpful comment, but we’re definitely getting the same issues as described above in production too.

For now we’re also restarting everything regularly with pm2.

Can’t share too much other info about our setup on here, but if someone from prisma wants to reach out I can spare some time.

So I can confirm that the reproduction outputs the following message:

gitpod /workspace/prisma-nestjs-memory-leak-repoduction (main) $ npm run test:leak

> prisma-memory-leak@0.0.0 test:leak
> node --expose-gc ./node_modules/.bin/jest --config=apps/prisma-memeory-leak/jest.config.ts --verbose --no-compilation-cache --detectLeaks --runTestsByPath ./apps/prisma-memeory-leak/src/app/app-prisma.controller.spec.ts

  console.log
    before prisma close

      at Proxy.<anonymous> (src/app/prisma/prisma.service.ts:6:13)

  console.log
    after prisma close

      at Proxy.<anonymous> (src/app/prisma/prisma.service.ts:8:13)

 FAIL   prisma-memeory-leak  apps/prisma-memeory-leak/src/app/app-prisma.controller.spec.ts
  ● Test suite failed to run

    EXPERIMENTAL FEATURE!
    Your test suite is leaking memory. Please ensure all references are cleaned.

    There is a number of things that can leak memory:
      - Async operations that have not finished (e.g. fs.readFile).
      - Timers not properly mocked (e.g. setInterval, setTimeout).
      - Keeping references to the global scope.

      at onResult (../../node_modules/@jest/core/build/TestScheduler.js:190:18)
      at ../../node_modules/@jest/core/build/TestScheduler.js:316:17
      at ../../node_modules/emittery/index.js:260:13
          at Array.map (<anonymous>)
      at Emittery.emit (../../node_modules/emittery/index.js:258:23)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        2.355 s
Ran all test suites within paths "./apps/prisma-memeory-leak/src/app/app-prisma.controller.spec.ts".
Segmentation fault

I also sent a PR to make the reproduction actually work out of the box: https://github.com/adrian-goe/prisma-nestjs-memory-leak-repoduction/pull/1

Note: Even though this is now on confirmed, it does not mean there has to be a problem at Prisma. We will keep investigating that.

Same - had a few weeks without issue since the upgrade, seems to be fixed!

I believe that we are also experiencing this - I can’t give source code publicly, but I am happy to add a few developers to our repo for research purposes if that helps. Nest.js - Node 18 - Prisma, deployed on Heroku, crashes out with heap allocation about once per day.

I wrote a very simple sample app and left it running over the weekend and I believe it repros the issue. The code is here: https://github.com/whalesync-ryder/prisma-leak-demo

Here’s the memory over 27 hours:

Let me know if you have any questions, hope that helps! Edit: forgot to mention running on node v16.13.0. (which we need to upgrade from but you know)

@GrinZero只是好奇，您能否详细说明您修改的内容以及您看到的效果？

Update somethings:

• NestJs ^8 => ^10
• Prisma ^3 => ^5
• Node ^16 =>^20

I use prom-client and grafana to monitor memory.

Before Update：

After Update：

@mmmeff I tested it with the reproduction, i posted. That did not fix it.

Node: 20.6.1 Prisma: 5.2.0

@janpio I pushed it on a seperate branch https://github.com/adrian-goe/prisma-nestjs-memory-leak-repoduction/tree/node20

edit: here is a PR with a test pipline: https://github.com/adrian-goe/prisma-nestjs-memory-leak-repoduction/pull/3

can’t tell you guys how excited i am to have finally found others in my boat, this mem leak has been plaguing my highly used app in prod. my prod server crashes every 2 days due to a memory leak, goes over 95% then poof on its own if i dont keep an eye on it. i have to restart it nightly. i have a cronjob that restarts the dota docker container for me nightly, which uses the most prisma, but my other packages still use prisma so those are more slowly leaking too and i’ll have to restart the whole server eventually (like today it crashed in 3 days despite restarting dota once a night)

after adding new commits that depend more on db queries, it now shoots up from ~350mb-400mb before a restart in 2 days, to ~650mb in just a few hours (master branch) !

last known “working” version with 350mb-450mb rss usage over a few days (still needs restarted, it goes up to 600mb+ if left alone)

• https://github.com/dotabod/backend/tree/81f939bfe7bc4b72e17e3fcc95128f0f9782268f

commits since then have created the memory leak within 3 hours to 650mb+. somewhere in here causes a big prisma memory leak i just haven’t inspected the commits to see what it is (i guessed the extra db calls)

• https://github.com/dotabod/backend/compare/8de0fde6815494d776ae7a91519885bf21197fea...4b41b182fc43ba08e5aaf73f6d27074755a11149
• everything after the above commits to master is just me trying to scramble and find the leak, adding unit tests to reproduce locally

reproduce: i can reproduce this locally every time using my unit tests, you can too if you clone & run the prisma seeder. feel free to add me as friend on discord #techleed if you need help setting up

• https://github.com/dotabod/backend

linkback to what i thought was the issue (overusing global variables), but now can see it’s prisma

• https://github.com/dotabod/backend/issues/339

i use prisma 5.2.0

• https://github.com/dotabod/backend/blob/master/packages/prisma/package.json
• https://github.com/dotabod/backend/blob/master/package.json

this is the package that crashes

• https://github.com/dotabod/backend/blob/master/packages/Dockerfile.dota
• https://github.com/dotabod/backend/blob/master/docker-compose.yml#L65

here’s the prisma schema, i use both mongo and psql

• https://github.com/dotabod/backend/blob/master/packages/prisma/src/postgres.prisma
• https://github.com/dotabod/backend/blob/master/packages/prisma/src/mongo.prisma

prod runs on

• Linux ec2.internal 6.1.34-59.116.amzn2023.x86_64 1 SMP PREEMPT_DYNAMIC Thu Jun 29 18:11:59 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

my local repro ran on (M1 Max)

• Darwi 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

Sounds like the next thing I need to try is prisma 4.8.1

(no sensitive data here, just seed data)

This issue is keeping us from upgrading past 4.8, since our containers die after around a day. There’s lots of features in 5.0 i’m looking on with envy until this gets resolved.

It’s worth noting that the existing memory leak tests will not detect this issue because they’re examining only the NodeJS heap size; it’s likely that the existing test suite could be adjusted to catch this class of issue by using process.memoryUsage().rss to capture total memory usage, rather than heapUsed here: https://github.com/prisma/prisma/blob/main/packages/client/tests/memory/_utils/createMemoryTest.ts#L63

So I can confirm that the reproduction outputs the following message:

gitpod /workspace/prisma-nestjs-memory-leak-repoduction (main) $ npm run test:leak

> prisma-memory-leak@0.0.0 test:leak
> node --expose-gc ./node_modules/.bin/jest --config=apps/prisma-memeory-leak/jest.config.ts --verbose --no-compilation-cache --detectLeaks --runTestsByPath ./apps/prisma-memeory-leak/src/app/app-prisma.controller.spec.ts

  console.log
    before prisma close

      at Proxy.<anonymous> (src/app/prisma/prisma.service.ts:6:13)

  console.log
    after prisma close

      at Proxy.<anonymous> (src/app/prisma/prisma.service.ts:8:13)

 FAIL   prisma-memeory-leak  apps/prisma-memeory-leak/src/app/app-prisma.controller.spec.ts
  ● Test suite failed to run

    EXPERIMENTAL FEATURE!
    Your test suite is leaking memory. Please ensure all references are cleaned.

    There is a number of things that can leak memory:
      - Async operations that have not finished (e.g. fs.readFile).
      - Timers not properly mocked (e.g. setInterval, setTimeout).
      - Keeping references to the global scope.

      at onResult (../../node_modules/@jest/core/build/TestScheduler.js:190:18)
      at ../../node_modules/@jest/core/build/TestScheduler.js:316:17
      at ../../node_modules/emittery/index.js:260:13
          at Array.map (<anonymous>)
      at Emittery.emit (../../node_modules/emittery/index.js:258:23)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        2.355 s
Ran all test suites within paths "./apps/prisma-memeory-leak/src/app/app-prisma.controller.spec.ts".
Segmentation fault

I also sent a PR to make the reproduction actually work out of the box: adrian-goe/prisma-nestjs-memory-leak-repoduction#1

Note: Even though this is now on confirmed, it does not mean there has to be a problem at Prisma. We will keep investigating that.

I’ve created a new issue where there’s a reproduction with no other dependency but Prisma.

• https://github.com/prisma/prisma/issues/18934

Technically I cannot call it a memory leak but I am for sure calling it a memory mismanagement.

My feeling is that the following commit is causing it: https://github.com/prisma/prisma/commit/f395abad8eb85739a1df35071caa9e5050993696 but that’s just a feeling since I know the prisma infrastructure is super complex and I’m not a Rust developer.

I hope it helps in your investigation.

Hey @livthomas we’ll need further information to be able to help you at all. Ideally, a minimal reproduction repro of the issue would be best (schema, js/ts code, query data). It would be greatly beneficial if you could share what exactly is going wrong, specific error messages, specific outcomes you’re running into, and if possible, any graphs / data that indicate a memory leak (or high memory usage). Here’s an example of what I mean per graphs / data for profiling such issues.

I am in the process of upgrading my node right now, so i should be able to confirm pretty soon if that resolved it for me. Thanks for the info.

I wrote a very simple sample app and left it running over the weekend and I believe it repros the issue. The code is here: whalesync-ryder/prisma-leak-demo

Here’s the memory over 27 hours:

Let me know if you have any questions, hope that helps! Edit: forgot to mention running on node v16.13.0. (which we need to upgrade from but you know)

@whalesync-ryder I could reproduce finally using Node.js v16, the reproduction in GitHub Actions was missing yarn prisma db push at first, so previous reproductions were actually not executing any queries, they were all failing because the database didn’t exist.

Did you upgrade to v18 or v20 already? This problem looks only related to Node.js v16 (which is “end of life”) to me at the moment, see results:

Note:

• I used --max-old-space-size to show how rss is actually reclaimed and growing rss is not a problem in itself, as long as it’s reclaimed when there is some memory pressure.

---

Using nest start

• Upgrade all + Prisma 5.5.2 + --max-old-space-size=150 - Node v16.20.2 https://docs.google.com/spreadsheets/d/14-9DFfkW_DD_dAdj3WTsQ4kTm_HONd0fgE1rgUgGljo/edit?usp=sharing 🔴 There is definitely a leak here!

• Upgrade all + Prisma 5.5.2 + --max-old-space-size=150 - Node v18.18.2 Actually ran out of memory at nest start, see next section for results

• Upgrade all + Prisma 5.5.2 + --max-old-space-size=150 - Node v20.9.0 https://docs.google.com/spreadsheets/d/1eSGaXS34aMqWJUW9r-oFjX5IU8ojQ5lAhSHHMsRJzQ4/edit?usp=sharing

---

Using yarn run build and then node dist/main

Notes:

• nest start appears to consume a lot of memory when starting, so I switched to building and starting the server with Node.js instead.

• Upgrade all + Prisma 5.5.2 + --max-old-space-size=150 + node dist/main - Node v16.20.2 https://docs.google.com/spreadsheets/d/1BQJLIMGl3xCQSFMNfYUgzdcAUhxaxHc8JnZqlFWJJJg/edit?usp=sharing 🔴 There is definitely a leak here!

• Upgrade all + Prisma 5.5.2 + --max-old-space-size=150 + node dist/main - Node v18.18.2 https://docs.google.com/spreadsheets/d/1rDRyzjWi15ZC4az2-Hz0mULcskx98xfQ8ODoAYkGV7E/edit?usp=sharing

• Upgrade all + Prisma 5.5.2 + --max-old-space-size=150 + node dist/main - Node v20.9.0 https://docs.google.com/spreadsheets/d/16hOzvpRsdtt3eay8xejz3ukTFByjPkm0t_6d1d_cFoY/edit?usp=sharing

I needed to run a script to import around 12 million records.

await db.record.findUnique({
    select: {
      id: true,
    },
    where: {
      id: key,
      updatedAt: { gte: lastModifiedDate },
    },
  })

Runing this query on Prisma 5.4.1 and NodeJS 18.12.0 was causing a memory leak. A simple script running only this query would consume all server memory and crash. Optimizing it with $queryRaw didn’t help. Then I tried calling the raw query with pg 8.11.3 on Nodejs 18.12.0 and got normal memory consumption so the problem was with Prisma 5.4.1 on NodeJS 18.12.0. I upgraded to NodeJS 20.8.0 and tested the original findUnique on Prisma 5.4.1 - this solved the problem. The server is running Debian GNU/Linux 10 (buster), but the same thing was happening on Mac OS X.

@janpio Very interesting! So you’re not seeing any difference. Maybe I had something else going on to explain the 10x difference in leak speed between my two runs. I can’t imagine what it would be.

@janpio I can try, will use the same setup from the main instance. we are currently using engineType = "library" on node:18.16-bullseye-slim, so not an ARM64 issue (as opposed to the other thread #18510). we enabled the pprof profiler on some % of the instances to see who the culprit is and so far seems to be prisma

same here, it’s a slow memory leak, and we’re restarting the containers after it crosses the 95% memory usage. it grows regardless of how much traffic the server is getting. all replicas eventually get OOM’d in a couple of hours (8-12 hours)

Just to give you an idea what this memory leak look like in another app when we keep it running for a day:

apps        83  6.8 19.6 21958328 205820 ?     Sl   Jul29 142:00 /layers/heroku_nodejs-engine/nodejs/bin/node build/worker.js
apps        94 10.7 51.3 1267276 538512 ?      Sl   Jul29 223:34 /workspace/node_modules/.prisma/client/query-engine-debian-openssl-1.0.x --enable-raw-queries --enable-metrics --enable-open-telemetry --port 0 --engine-protocol json

This app is a job executor that works all the time without any break since there are always more jobs than it can process. And the Node.js app itself still consumes just 196 MB of RAM while Prisma query engine consumes 513 MB which is constantly growing until the container eventually runs out of memory and is restarted.

for now I think using JavaScript only client and engine is the solution,

or move to Prisma for Schema + Kysely for data fetch and migration because Prisma not good enough

We’re also seeing a very clear memory leak in prisma, staring somewhere between version 4.3.1 and 4.12.0. The amount of leaked memory seems to depend on the amount of queries made with prisma, with some of our high-load services running out of memory in a day or two.

@xlmnxp since this is also a problem in our CI, that don’t use arm64, it is probably not related to this.

Hmm, we have same issue, it was working fine, but I don’t know what break