prisma: `updateMany` memory leak?

Bug description

Description

I’m using MongoDB, and running a large number of updates when using Prisma will cause memory leakage.

Total number of data

Number of data: 9 400 000

const client = new MongoClient(DATABASE_URL_DEVELOPMENT);
await client.connect();
const db = client.db();
const collection = db.collection('test');
const totalDocuments = await collection.countDocuments();
console.log(totalDocuments);

스크린샷 2023-12-12 200502

Issues

It works slowly but normally in mongodb. But when you update a lot on Prima, your RAM gets higher and higher.

image) https://ibb.co/album/k5B0RF sort A/Z

How to reproduce

Expected behavior

No response

Prisma information

model Test {
  @@map("test")

  objectId String @id @default(auto()) @map("_id") @db.ObjectId
  id              String   @unique
  na              String
  sa              String
  
  ap              Float
  co              String
  eq              Float
  oi              String
  oq              Float
  ot              String
  pr              Float
  ro              Boolean
  si              String
  ps              String
  st              String
  sp              Float?
  cp              Boolean
  sy              String
  ti              Int
  tf              String
  ty              String
  ac              Float?
  pr2              Float?
  ut              Int
  wt              String
  pp              Boolean


  pi              String?

  ic              Boolean  @default(false)
  ca              DateTime? @default(now())
  ua              DateTime? @updatedAt
  da              DateTime?
}

 const result = await prisma.test.updateMany({
    data: {
      ic: false,
    },
  });
  console.log(result);

-> leak

Environment & setup

OS: WSL2 Ubuntu (host is windows)
Database: MongoDB
Node.js version: v18.17.1

WSL

WSL version: 2.0.9.0 Kernel version: 5.15.133.1-1 WSLg version: 1.0.59 MSRDC version: 1.2.4677 Direct3D version: 1.611.1-81528511 DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp Windows version: 10.0.22631.2792

lsb_release -a

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy

cat /etc/issue

Ubuntu 22.04.3 LTS

Prisma Version

prisma                  : 5.7.0
@prisma/client          : 5.7.0
Computed binaryTarget   : debian-openssl-3.0.x
Operating System        : linux
Architecture            : x64
Node.js                 : v18.17.1
Query Engine (Node-API) : libquery-engine 79fb5193cf0a8fdbef536e4b4a159cad677ab1b9 (at ../../node_modules/.pnpm/@prisma+engines@5.7.0/node_modules/@prisma/engines/libquery_engine-debian-openssl-3.0.x.so.node)
Schema Engine           : schema-engine-cli 79fb5193cf0a8fdbef536e4b4a159cad677ab1b9 (at ../../node_modules/.pnpm/@prisma+engines@5.7.0/node_modules/@prisma/engines/schema-engine-debian-openssl-3.0.x)
Schema Wasm             : @prisma/prisma-schema-wasm 5.7.0-41.79fb5193cf0a8fdbef536e4b4a159cad677ab1b9
Default Engines Hash    : 79fb5193cf0a8fdbef536e4b4a159cad677ab1b9
Studio                  : 0.495.0

MongoDB Server

OS: Windows10 22H2(19045.3693) MongoDB Version: 6.0.5.0 (64bit)

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 16 (11 by maintainers)

Most upvoted comments

@Jolg42 I’ll check. thank you

somersby10ml on Dec 14, 2023

@Jolg42

Why are you running multiple processes of the same command at the same time?

This is not what I meant. It seems to have been automatically divided by node.

package.json

 "script": "cross-env $(node env.js) ts-node -r tsconfig-paths/register --files",

When I did the test, I did the following.

pnpm dotenv -- pnpm script server -- ./test/migration/prisma2.ts

What’s in that script exactly? Only one Prisma query?

Yes

async function main() {
  const prisma = new PrismaClient({
    datasources: {
      db: {
        url: DATABASE_URL_DEVELOPMENT,
      },
    },
  });
  const result = await prisma.test.updateMany({
      data: {
        ic: false,
      },
    });
}
main();

Is it running the exact same query…

Yes

It looks like this query would be resource intensive…

I tried adding an index, but the same symptoms appear.

somersby10ml on Dec 12, 2023

So for the 2 errors:

Connection reset by peer (os error 104)
- it sounds to me that the MongoDB server restarted and the connection was reset
  - I think this should be visible in logs, did you see anything about this?
An established connection was aborted by the software in your host machine.(os error 10053)
- Here is some documentation from Microsoft https://learn.microsoft.com/en-us/windows/win32/winsock/windows-sockets-error-codes-2
WSAECONNABORTED 10053 Software caused connection abort. An established connection was aborted by the software in your host computer, possibly due to a data transmission time-out or protocol error.

Other things you could check is to change the default settings:

for maxTimeMS, it looks like the default is 60 seconds. See https://www.mongodb.com/docs/compass/current/agg-pipeline-builder/maxtime-ms-pipeline/ it shows how to do it from the Compass App. I think you can also set this via the connection string.
wtimeoutMS should be fine by default (no timeout), see docs
connectTimeoutMS is set by default to 10 seconds. (see code)
- I don’t think it’s relevant here, though you could put a high value to make sure it’s not a problem.

For more details, check https://www.mongodb.com/docs/drivers/node/current/faq/#what-is-the-difference-between--connecttimeoutms----sockettimeoutms--and--maxtimems--

My theory at the moment is that your query is too big, which either ends up in a time-out or causes excessive memory usage and ends up with these errors as the MongoDB server is not able to process them. Probably because your OS kills the process and restarts it.

How much RAM do you have on that machine? (and how much is free?)
Try changing the settings, especially maxTimeMS

Jolg42 on Dec 14, 2023

@Jolg42 more test

Windows The current connection has been interrupted by software on your host system.

WSL Ubuntu: Connection reset by peer (os error 104) or ELIFECYCLE Command failed with exit code 1.

MacOS(ram 16GB): 제목 없음

The point is that when there is a lot of data, running updateMany increases RAM usage and terminates the program.

somersby10ml on Dec 14, 2023

@Druue

Which version of MongoDB are you using?

I updated the bottom of the question.

I’m using pnpm

The same symptoms appear in the latest version of prisma (5.7.0).

Memory does not return to a certain level, continues to grow, and eventually the program ends.

mongodb’s updateMany works normally. However, it seems that Prisma updateMany ends without completing it until the end. https://ibb.co/dr4Zcys

somersby10ml on Dec 12, 2023