prisma: Prisma generate randomly fails on Ubuntu due to missing internal .so `libquery_engine-debian-openssl-1.1.x.so.node`

We are on version 5.5.2, we run multiple generate in parallel with a command yarn workspaces foreach --parallel prisma generate and the CI on GitHub is still flaky and fails regularly with a similar error: Error: ENOENT: no such file or directory, stat '/home/runner/work/spendesk/spendesk/node_modules/prisma/libquery_engine-debian-openssl-1.1.x.so.node'.

We managed to pinpoint the issue to a possible race condition when running multiple prisma generate commands in parallel. Setting DEBUG='prisma*' revealed that both schema-1 and schema-2 try to download and write to the same file at the same time.

schema-1 prisma:generate: 2023-08-25T12:25:08.238Z prisma:fetch-engine:download  Downloading https://binaries.prisma.sh/all_commits/2804dc98259d2ea960602aca6b8e7fdc03c1758f/debian-openssl-1.1.x/libquery_engine.so.node.gz to /home/runner/work/pratiq-microservices/pratiq-microservices/node_modules/prisma/libquery_engine-debian-openssl-1.1.x.so.node 
schema-2 prisma:generate: 2023-08-25T12:25:08.186Z prisma:fetch-engine:download  Downloading https://binaries.prisma.sh/all_commits/2804dc98259d2ea960602aca6b8e7fdc03c1758f/debian-openssl-1.1.x/libquery_engine.so.node.gz to /home/runner/work/pratiq-microservices/pratiq-microservices/node_modules/prisma/libquery_engine-debian-openssl-1.1.x.so.node 

(Note: this only occurs if the engine binaries do not exist and need to be downloaded)

Changing our script to run prisma generate for each schema sequentially instead of in parallel seems to reliably fix the issue.

Note: This is potentially leading to problems on Ubuntu that we are currently investigating. We might need to revert. @Jolg42 is on this.

Thanks, I ran our CI three times with the new release and it stays green!

@lukahartwig & others: You can now try out this 5.5.0-dev.45 dev version. Like npm install --dave-dev prisma@5.5.0-dev.45 && npm install @prisma/client@5.5.0-dev.45

Let us know if that makes things better 🙌🏼

Note that the official release, 5.5.0 is planned for October 24th.

now the bug fairy just needs to put a PR under my pillow 😃

Just got the error again, here are the debug logs:

  > DEBUG=prisma* prisma generate --schema=prisma/legacy.prisma
  
  2023-07-29T11:19:59.391Z prisma:engines  binaries to download libquery-engine, schema-engine
  2023-07-29T11:19:59.395Z prisma:get-platform  Found distro info:
  {
    "targetDistro": "debian",
    "familyDistro": "debian",
    "originalDistro": "ubuntu"
  }
  2023-07-29T11:19:59.395Z prisma:get-platform  Trying platform-specific paths for "debian" (and "ubuntu")
  2023-07-29T11:19:59.397Z prisma:get-platform  Found libssl.so file using platform-specific paths: libssl.so
  2023-07-29T11:19:59.397Z prisma:get-platform  The parsed libssl version is: undefined
  2023-07-29T11:19:59.397Z prisma:get-platform  Falling back to "ldconfig" and other generic paths
  2023-07-29T11:19:59.435Z prisma:get-platform  Command "ldconfig -p | sed "s/.*=>s*//" | sed "s|.*/||" | grep libssl | sort | grep -v "libssl.so.0"" successfully returned "libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  "
  2023-07-29T11:19:59.435Z prisma:get-platform  Found libssl.so file using "ldconfig" or other generic paths: libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  
  2023-07-29T11:19:59.435Z prisma:get-platform  The parsed libssl version is: 1.1.x
  2023-07-29T11:19:59.441Z prisma:get-platform  Found distro info:
  {
    "targetDistro": "debian",
    "familyDistro": "debian",
    "originalDistro": "ubuntu"
  }
  2023-07-29T11:19:59.441Z prisma:get-platform  Trying platform-specific paths for "debian" (and "ubuntu")
  2023-07-29T11:19:59.442Z prisma:get-platform  Found libssl.so file using platform-specific paths: libssl.so
  2023-07-29T11:19:59.442Z prisma:get-platform  The parsed libssl version is: undefined
  2023-07-29T11:19:59.442Z prisma:get-platform  Falling back to "ldconfig" and other generic paths
  2023-07-29T11:19:59.464Z prisma:get-platform  Command "ldconfig -p | sed "s/.*=>s*//" | sed "s|.*/||" | grep libssl | sort | grep -v "libssl.so.0"" successfully returned "libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  "
  2023-07-29T11:19:59.464Z prisma:get-platform  Found libssl.so file using "ldconfig" or other generic paths: libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  
  2023-07-29T11:19:59.464Z prisma:get-platform  The parsed libssl version is: 1.1.x
  2023-07-29T11:19:59.924Z prisma:loadEnv  project root found at /home/runner/work/gw2treasures.com/gw2treasures.com/apps/legacy-importer/package.json
  2023-07-29T11:19:59.931Z prisma:tryLoadEnv  Environment variables loaded from /home/runner/work/gw2treasures.com/gw2treasures.com/apps/legacy-importer/.env
  Environment variables loaded from .env
  Prisma schema loaded from prisma/legacy.prisma
  2023-07-29T11:19:59.953Z prisma:get-platform  Found distro info:
  {
    "targetDistro": "debian",
    "familyDistro": "debian",
    "originalDistro": "ubuntu"
  }
  2023-07-29T11:19:59.953Z prisma:get-platform  Trying platform-specific paths for "debian" (and "ubuntu")
  2023-07-29T11:19:59.955Z prisma:get-platform  Found libssl.so file using platform-specific paths: libssl.so
  2023-07-29T11:19:59.955Z prisma:get-platform  The parsed libssl version is: undefined
  2023-07-29T11:19:59.955Z prisma:get-platform  Falling back to "ldconfig" and other generic paths
  2023-07-29T11:19:59.984Z prisma:get-platform  Command "ldconfig -p | sed "s/.*=>s*//" | sed "s|.*/||" | grep libssl | sort | grep -v "libssl.so.0"" successfully returned "libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  "
  2023-07-29T11:19:59.984Z prisma:get-platform  Found libssl.so file using "ldconfig" or other generic paths: libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  
  2023-07-29T11:19:59.984Z prisma:get-platform  The parsed libssl version is: 1.1.x
  2023-07-29T11:19:59.993Z prisma:get-platform  Found distro info:
  {
    "targetDistro": "debian",
    "familyDistro": "debian",
    "originalDistro": "ubuntu"
  }
  2023-07-29T11:19:59.993Z prisma:get-platform  Trying platform-specific paths for "debian" (and "ubuntu")
  2023-07-29T11:19:59.994Z prisma:get-platform  Found libssl.so file using platform-specific paths: libssl.so
  2023-07-29T11:19:59.994Z prisma:get-platform  The parsed libssl version is: undefined
  2023-07-29T11:19:59.994Z prisma:get-platform  Falling back to "ldconfig" and other generic paths
  2023-07-29T11:20:00.017Z prisma:get-platform  Command "ldconfig -p | sed "s/.*=>s*//" | sed "s|.*/||" | grep libssl | sort | grep -v "libssl.so.0"" successfully returned "libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  "
  2023-07-29T11:20:00.017Z prisma:get-platform  Found libssl.so file using "ldconfig" or other generic paths: libssl.so
  libssl.so.1.1
  libssl.so.3
  libssl3.so
  
  2023-07-29T11:20:00.017Z prisma:get-platform  The parsed libssl version is: 1.1.x
  2023-07-29T11:20:00.110Z prisma:download  copying /home/runner/.cache/prisma/master/6b0aef69b7cdfc787f822ecd7cdc76d5f1991584/debian-openssl-1.1.x/libquery-engine to /home/runner/work/gw2treasures.com/gw2treasures.com/node_modules/prisma/libquery_engine-debian-openssl-1.1.x.so.node
  node:events:492
        throw er; // Unhandled 'error' event
        ^
  
  Error: ENOENT: no such file or directory, open '/home/runner/work/gw2treasures.com/gw2treasures.com/node_modules/prisma/libquery_engine-debian-openssl-1.1.x.so.node'
  Emitted 'error' event on ReadStream instance at:
      at emitErrorNT (node:internal/streams/destroy:151:8)
      at emitErrorCloseNT (node:internal/streams/destroy:116:3)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    errno: -2,
    code: 'ENOENT',
    syscall: 'open',
    path: '/home/runner/work/gw2treasures.com/gw2treasures.com/node_modules/prisma/libquery_engine-debian-openssl-1.1.x.so.node'
  }
  
  Node.js v20.5.0
  npm ERR! Lifecycle script `prisma:generate` failed with error: 
  npm ERR! Error: command failed 
  npm ERR!   in workspace: legacy-importer 
  npm ERR!   at location: /home/runner/work/gw2treasures.com/gw2treasures.com/apps/legacy-importer 

This was the first time after maybe 30 - 40 CI runs, it still feels like enabling the debug output improved things…

This issue is really becoming painful for our team. Ruins most CI runs

winrar

I just tested wrapping the line in an os.platform() === "darwin" check and ran our CI a couple of times. This seems to work. Should I create a PR?

The problem happens on Ubuntu on GitHub Actions for us. We don’t usually run Prisma in parallel locally so making an exception for non-Mac platforms would improve our situation.

I ran into this issue as well and I think the issue is in this line.

We are running multiple prisma generate processes in a monorepo using pnpm workspaces and turborepo. Setting the concurrency to 1 in turborepo resolves the issues.

It looks like the prisma cli deletes and replaces files but not in a concurrency safe way. So other processes that expect those files to be there fail on random fs calls.

Got the other error case as well now (by running while clear && find . -type d -name "node_modules" -exec rm -rf {} + && pnpm install; do :; done in my reproduction repository until I got a different error):

│ packages/1 copy 9 prisma-generate: Environment variables loaded from .env
│ packages/1 copy 9 prisma-generate: Prisma schema loaded from prisma/schema.prisma
│ packages/1 copy 2 prisma-generate: Error: ENOENT: no such file or directory, stat '/workspace/prisma-19124/node_modules/.pnpm/prisma@5.3.0/node_modules/prisma/libquery_engine-debian-openssl-3.0.x.so.node'
│ packages/1 copy 2 prisma-generate: Failed
│ undefined
│ /workspace/prisma-19124/packages/1 copy 2:
│  ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  1@1.0.0 prisma-generate: `prisma generate`
│ Exit status 1
└─ Failed in 19.3s at /workspace/prisma-19124
 ELIFECYCLE  Command failed with exit code 1.

I think that covers all errors that have been mentioned in this issue.

I know, that was meant about how to reproduce the error - try again until it happens in the reproduction repository. It does not happen that often to me.

I did not run into this issue over the last 2 weeks since making sure that prisma generate is not run in parallel anymore.

I’ve setup my generate tasks as dependencies in turborepo, so they should build in sequence. I will report back if the issue is gone over the next few days…

Perhaps, but @janpio we use turborepo which implicitly paralellizes actions. would have to do some acrobatics to set up everything sequentially

@Jolg42 its very flakey. As @darthmaim mentioned, I get this sometimes 5 times in a row. Sometimes 20 times without occurring.

I’ve added DEBUG=prisma* to my CI script.

I restarted the job 10 times and didn’t get a single error, usually I only get 3 runs or so in a row without the error… So either I’m really lucky (or unlucky), or the additional debug logs somehow prevent a race condition. I’ve hit the error multiple earlier today before adding the debug logs.

I will report back when the error comes back with some additional logs