prisma: Segmentation fault crash when using prisma client when using PostgreSQL
Bug description
When trying to use findMany() (or any other method) on a model, prisma client crashes and the following line appears in my log:
Segmentation fault (core dumped).
This only happens if I use PostgreSQL as far as I know, this first appeared when I switched to PostgreSQL from mysql.
How to reproduce
- Initiate a new prisma project (make sure to use PostgreSQL) and generate the client - no errors there
- Write a simple client which fetches some data from the databse
- The script crashes and
Segmentation fault (core dumped)gets logged in the console
Expected behavior
The script does not crash.
Prisma information
Example schema:
generator client {
provider = "prisma-client-js"
}
datasource db {
provider = "postgresql"
url = env("DATABASE_URL")
}
model User {
id String @id
email String?
}
Example test client:
const {PrismaClient} = require('@prisma/client');
const prisma = new PrismaClient({ log: ['query', 'info', 'warn'] })
prisma.user.findMany().then((e) => {
console.log(e)
})
Environment & setup
- OS: Pop!_OS (
Linux pop-os 5.15.5-76051505-generic #202111250933~1638201579~21.04~09f1aa7-Ubuntu SMP Tue Nov 30 02: x86_64 x86_64 x86_64 GNU/Linux) - Database: PostgreSQL v14.1 (Ubuntu 14.1-1.pgdg21.04+1)
- Node.js version: v17.2.0
Prisma Version
prisma : 3.6.0
@prisma/client : 3.6.0
Current platform : debian-openssl-1.1.x
Query Engine (Node-API) : libquery-engine dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/libquery_engine-debian-openssl-1.1.x.so.node)
Migration Engine : migration-engine-cli dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/migration-engine-debian-openssl-1.1.x)
Introspection Engine : introspection-core dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/introspection-engine-debian-openssl-1.1.x)
Format Binary : prisma-fmt dc520b92b1ebb2d28dc3161f9f82e875bd35d727 (at node_modules/@prisma/engines/prisma-fmt-debian-openssl-1.1.x)
Default Engines Hash : dc520b92b1ebb2d28dc3161f9f82e875bd35d727
Studio : 0.440.0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 47
- Comments: 113 (34 by maintainers)
Commits related to this issue
- Force node v16 @see https://github.com/prisma/prisma/issues/10649#issuecomment-1022699863 — committed to diglactic/scheduler by shengslogar 2 years ago
- Downgrade node due to prisma segfault on supabase Ref: `https://github.com/prisma/prisma/issues/10649#issuecomment-1247961614` — committed to arilence/icalendar-trimmer by arilence a year ago
- chore(Retire prisma): Removed prisma following a SegFault Error. - https://github.com/prisma/prisma/issues/10649. - switched to @fastify/postgres. - all statuses are presently in string form, will sw... — committed to grassrootseconomics/cic-ussd by deleted user a year ago
- fix(api): prisma segfault https://github.com/prisma/prisma/issues/10649 — committed to vladfaust/aistories by vladfaust a year ago
- Prisma seed is segfaulting. Upgrade prisma and @prisma/client to try to fix. See https://github.com/prisma/prisma/issues/10649#issuecomment-1480051566 Upgrade to 4.12.0-integration-rtld-deepbind.3 — committed to navapbc/wic-participant-recertification-portal by rocketnova a year ago
- fix(engine-core): use RTLD_DEEPBIND flag when loading the addon (#18426) Fixes: #10649 Fixes: #17946 Fixes: #16897 Fixes: #17223 Fixes: #18336 Fixes: #18559 — committed to prisma/prisma by aqrln a year ago
- add engine type to schema see https://github.com/prisma/prisma/issues/10649 — committed to mint-vernetzt/community-platform by phollome 9 months ago
- fix: Unable to run "remove is priviledge" migration (#1027) see https://github.com/prisma/prisma/issues/10649 — committed to mint-vernetzt/community-platform by phollome 9 months ago
While we are looking for the best way to fix or prevent the underlying problem, the recommended workaround for affected users is to make sure that the versions of OpenSSL used by Node.js and the system OpenSSL match:
If your distribution provides a build of Node.js 18 with OpenSSL 1.1.1, you can use that instead of the upstream Node.js build that vendors OpenSSL 3. This is the case on Alpine, where you can install
nodejs-currentfrom repositorycommunity: it provides (at the moment of writing) Node.js 18.6.0 linked with OpenSSL 1.1.1q, and the issue is not reproducible when using it.Alternatively, if you can’t do any of that and you need to stay with the upstream build of Node.js 18 on a system with OpenSSL 1.1.x, you can switch to the binary engine in Prisma.
Reproduction
Create a cloud PostgreSQL database. I used Amazon RDS PostgreSQL 14.3 and Supabase.
Create a minimal Prisma project. I used the following (but it doesn’t matter — you can reuse whatever test project you have):
Run it on x86-64 Debian 11 with Node.js 17+.4.
Process crashes with segmentation fault.
Run
ulimit -c unlimitedbeforehand if necessary if you want to get the core dump.Here’s the full backtrace with symbols: (click to expand)
And here’s the relevant part:
Immediate Cause
Whether the issue is reproducible depends on Node.js version: on the same system the reproduction script succeeds on Node.js 16.17.0 and older, but crashes with segmentation fault on Node.js 17.0.0 or newer.
Git bisect shows that the specific commit in Node.js that starts failing is https://github.com/nodejs/node/commit/66da32c045035cf2710a48773dc6f55f00e20c40.
This means that somehow the issue is triggered due to Node.js itself using bundled OpenSSL 3.0 and us using system OpenSSL, which in this case is 1.1.1n.
The specific instruction where it crashes inside OpenSSL:
The source code of the function:
In other words, the segfault happens due to the
->peeraccess on ther = s->session->peerline.Let’s check
s->sessionvalue:This indeed looks wrong: the value is oddly specific (looks like it came from some
intthat was equal to-1) and is definitely outside the heap (for comparison,sis0x7f5c88003330).Running this on ARM64 and putting a breakpoint there, you can see that the pointer looks normal, so it doesn’t just work “accidentally”, the bug indeed doesn’t occur on that platform.
Duplicate issues
Internal document
https://www.notion.so/prismaio/Segmentation-fault-with-cloud-PostgreSQL-b1b34ba631c44e9e9031f0cf63bcb23f
I’m using Ubuntu, and I was able to make it work by downgrading to
v16of Node. But this needs to be fixed. The other thing that I tried, and it worked (withNode v18), was addingengineType = "binary"toschema.prismaand runningnpx prisma generateagain.Add the same issue running on latest node version 17.4.0. Fixed by using the current LTS 16.3.2.
In my case, downgrading to
node 16.17.0(usingnode:16-bullseye) works. I did not change anything else!This is only a short term solution though.
This issue is fixed in Prisma 4.13.0 which will be released tomorrow, on Tuesday, Apr 18. If you need to temporarily use a pre-release version until then, you can use
4.13.0-dev.33.Note that this GitHub issue is about a specific bug that occurs when using Node.js 17+ on a system with OpenSSL 1.1 on x86-64 CPUs when using TLS to connect to database. Please do not post comments here if you encounter a segfault under different circumstances, and open a new issue instead!
If you are facing segfaults on ARM, please leave a comment in https://github.com/prisma/prisma/issues/18510 or open a new issue.
Hi everyone! Could you check if updating
prismaand@prisma/clientto4.12.0-integration-rtld-deepbind.3fixes the issue for you? It does in the cases that I tested, but it would be very helpful to test this on more different systems.This is an experimental dev version not intended for production but if it works as expected, we’ll work on making this land in a stable release. Thank you!
I’ve spent many hours trying to figure out why Prisma is segfaulting in the query engine, have tried changing the engine format to “engine” but without any promising result unfortunately.
I tried playing around with different versions and found that node 18.16 and above has this problem, from what I can tell node version
18.15is the latest version that runs without encountering any segfault. Worth noting is that the faulty versions did still work for smaller queries but they broke when GraphQL concatenated larger queries which I had a test case for.So basically this was the workaround I came up with:
@aqrln I tested this and it works:
(node 18, debian bullseye, hosted postgres DB, prisma 4.11.0) => error with segfault on connect
(node 18, debian bullseye, hosted postgres DB, prisma 4.12.0-integration-rtld-deepbind.3) => successful connection
@janpio Here are the steps to reproduce this. I used a fresh VM (Ubuntu 20.04 LTS) on Google Compute Engine. I used a fresh PostgreSQL database (try Supabase for a quick test).
schema.prisma:Now try and view a model on localhost:5555…
Prisma Studio frontend logs:
Console logs:
Error also seen with Ubuntu 18.04 LTS on Windows WSL2. I cannot reproduce this on my home setup (Arch Linux).
Error could not be fixed by changing engine to binary (cc @tisuela)
To anyone else using
linux-muslas their platform, upgrading prisma versions wasn’t enough for me. I also had to addengineType = "binary"to my client in theschema.prismafileJust deployed my app first time yesterday. Had the exact same issue as described here, literally tried everything before finding this thread. Fixed it by downgrading node:18 in my Dockerfile to node:16.18.0
With node:18 going into active LTS 6 days ago, stuff will hit the fan in here real soon I believe 😂 Especially as this setup is super common: Prisma, Node 18, standard DigitalOcean Droplet
My setup was:
DB: PSQL on Supabase
Node: node:18 downgrade to node:16.18.0
NestJS
Prisma: ^3.15.2
Production: DigitalOcean droplet with image: Ubuntu Docker 19.03.12 on Ubuntu 20.04
Local: M1 MacBook (had to adapt Dockerfile and prisma.schema’s binaryTargets to make it work on M1, doesn’t affect prod)
I still don’t know what the issue could be, but I have followed this guide to debug this issue a little bit more. I have to admit, I don’t have any experience with this, but I have attached crash.log file which got generated by the segfault-handler module:
Maybe this helps
@aqrln This works for me as well!
Node 18, Ubuntu 20.04 on Windows (WSL2), hosted PostgreSQL database (on GCP)
prismaand@prisma/clientversion 4.9 => Segmentation Fault:prismaand@prisma/clientversion 4.12.0-integration-rtld-deepbind.3 => works perfectly!:I hope you get the PR merged soon!
I’m using version 3.15.1 of Prisma. The error also happens when I specify
node:18.3.0-alpine. I’m using a Postgres database provided by Supabase.I seem to be having similar problems with the
node:18-alpinebase image in Docker. I get the error<container name> exited with code 139, which indicates a segmentation fault, as soon as my application callsfindMany.node:16-alpineworks. I’d like to use Node 18 if possible, though.Not sure if this helps but I tried using Google Cloud SQL for PostgreSQL but having the same issues:
Error persists when using version 3.5.0 for prisma CLI and client:
I just fixed this by using the binary engine instead of the library engine. Will there ever be a proper fix for this?
@aqrln Fixed here too 💯 same configuration as @mmachatschek
FWIW we are having the same issue.
This works fine locally Prisma:
4.2.0Node:18-slimPostgres (local which works fine):postgres:14.4-alpineThe same docker image that works locally vs a containerized postgres fails when deployed to ECS Fargate vs an RDS postgres db running
14.2with the same segmentation fault experienced by others.Found a solution that worked for me based on the info in this thread so I figured I should share.
I have been doing local development with no issues using docker for two months with no issues using the following versions: postgres: 14 node:18 prisma: 3.14.1
However, once i tried to deploy on AWS I got the segfault crash others in this thread were talking about. Our solution was to downgrade to use the following versions to get things working in AWS: Postgres: 13.4 node: 18 prisma: 4.1.0
Since there were no issue running the application in local containers, it seems like it is an issue with node version above 16 in combination with postgres 14 in cloud providers like AWS and GCP.
I’m having similar issue. But I can’t create a reproducible case, it only happens when running our whole application. I can only recreate it on ARM64 machine and not on AMD64 which is strange.
I have tried the new testing version, but it doesn’t fix my issue. More information can be found in the following slack thread https://prisma.slack.com/archives/CA491RJH0/p1679654874088859
Can someone facing this issue send us the core dump file? We would really like to look into this!
Just to add on here:
We are running in Cloud Run with Cloud SQL. We had a number of issues using the generally documented CloudSQL volume mounts to connect our Cloud Run instances to our database.
Switching to the IP address method in our DB Connection strings would cause our applications to segfault immediately.
The fix we found worked was to add
sslmode=disablewhen using direct IP addresses in our connection strings, and that seems to have resolved our issues. Just in case this helps anyone else who is having issues with Prisma and CloudRun/CloudSQL.We didn’t attempt any of the experiments above to change our binary type, or our nodejs version.
Downgrading the node version to 16 from 18 worked for me. It didn’t matter whether I set the engineType to binary or not. It worked in both cases. However, it didn’t work in node v18, even with binary engineType. I am using PopOs 22.04
I was having the same issue of
segmentation fault, interesting thing was, I was able to access db usingprisma studiobut not withprisma client, solved it by downgrading to node v16 from v18Getting segfaults as well with the new prisma version
4.10.1- we’re runningnode:16.16-alpine3.16- previously had to custom build an image using 4.8.1If you’re trying to run this inside the standard
node:18docker image then it’s not enough to just do:because the system includes
openssl@1.1.1by default.One way to get around this is to use
node:18-alpinewhich does not come withopensslinstalled out of the box. Then you can run:This will make the system use
openssl@3.xwhich will then mean that when the Prisma database client dynamically links to it, the right version is available. Et voila, no more segfault 😄Seems like the real issue here is that the Prisma database engine binary and Node binary link to
opensslin different ways which can cause version mismatches?TLDR;
✅
❌
Some additional details from newly added duplicate issues:
In https://github.com/prisma/prisma/issues/13641 @maximm98 reported the issue on RHEL (Rocky Linux 8). They say for them the issue started happening only after upgrading from Node.js 18.1.0 to Node.js 18.3.0, but I can’t reproduce it working with 18.1.0 — for me it crashes too.
In https://github.com/prisma/prisma/issues/15052#issuecomment-1239812483 @garth reported that the issue happens with PostgreSQL 14, but not with PostgreSQL 12. I haven’t try reproducing it, but it sounds like an interesting detail. Although it’s also possible that v12 was self-hosted and v14 is in cloud — the report doesn’t include that information.
Thank you, this is very valuable information.
No worries. Just don’t share it via public channels if you manage to get one later if it comes from the production environment — sorry if this is obvious, just wanted to point this out just in case.
I’ll try to reproduce it on Monday with the information we now have.
Hi, new to the party but having the same problem. Everything works locally and in our CI, but as soon as the code is deployed to AWS ECS (fargate), then it explodes in a fiery segfault 😃 I have a fresh dump of the trace for the segfault.
Docker image is built
FROM node:18-bullseye.Amazon RDS reports that the database in question is an
Aurora PostgreSQL version 14.3.Prisma is version
4.3.0And my
schema.prisma…I have tried setting
engineType = "binary", but that makes my CI pipelines stall forever. My next step is to try to downgrade node and if that does not work, downgrade the DB.Edit: for those interested I “generated” the segfault trace using segfault-handler, which will output the trace to both a file and stdout.
@aqrln
About the other things, see #47416
@AlexandreSenpai @conjohnerton Please do try https://github.com/prisma/prisma/issues/10649#issuecomment-1480051566 and let us know if this works for you.
Same for me: Running
npm i prisma@4.12.0-integration-rtld-deepbind.3did the trick!Hey guys, also faced this
Segmentation Faultwhen trying to connect to a Neon db inside a Github action. Using4.12.0-integration-rtld-deepbind.3version fixed it ✅ .Also had this issue on prisma
4.12.0. Switching over tobinaryengine in the client fixed it for now. Would love to use the default library, but not super important as of now 😃Thanks for looking into this, I saw that a change was potentially in flight already.
Node version:
v18.15.0Prisma version:v4.12.0@aqrln I create a new issue #18510
@onepunchcat looking at your
prisma -voutput, you’re on4.12.0-integration-views-fs.1and not4.12.0-integration-rtld-deepbind.3, could you please try again with4.12.0-integration-rtld-deepbind.3?Make sure to use an exact version in
package.jsonwithout a^in the beginning so that the package manager won’t grab something unrelated that was published under theintegrationtag later.For me the error changed when I change the engineType to
binary.This only occurred on side by side query is runing like:
But during multiples request, even without
Promise.allstill happens. Maybe it is a concurrently problem. Just in alpine, in a full image (FROM node:18) of node it is ok.Sure thing
binaryTargets = ["native", "linux-musl"]and adding OpenSSL to your DockerfileRUN apt-get update && apt-get install -y opensslIssue mentioned here: https://github.com/prisma/prisma/issues/8478#issuecomment-1095544159
FYI: If your distribution provides a build of Node.js 18 with OpenSSL 1.1.x, you can use that instead of the upstream Node.js build that vendors OpenSSL 3. This is the case on Alpine, where you can install
nodejs-currentfrom repositorycommunity: it provides (at the moment of writing) Node.js 18.6.0 linked with OpenSSL 1.1.1q, and the issue is not reproducible when using it.(added this to the post above as well)
Like @zanzlender, I found that it when using node v17.9 and trying to connect to supabase, I was getting the segmentation fault. However, since I have switched to the LTS (at the time of this writing, it is v16.15.x) it works.
I have the issue in nodes v18.1 after switching from AWS Aurora compatible Postgres to standard PostgreSQL on AWS. It worked for a few hours today on node v16.15; then, it started throwing this error every time I ran a query.
Windows 11 & WLS2 Ubuntu
Edit: Node v16.15.0 works - I ran
nvm set v16, and it only sets the version for the current terminal session. I set node to use v16 by default vianvm alias default 16.Note here that Node.js v17 is not recommended because not LTS yet. https://nodejs.org/en/about/releases/
I would be curious to know if you can reproduce with Node.js v16, current LTS
If this is failing on older Node versions, we can hopefully reproduce with that and then figure out what is going on.
So far the messages on WSL2 have been
Segmentation fault, and the messages on a Ubuntu VM have beenSegmentation fault (core dumped)Let’s close this then 👍
@sveisvei If Prisma not working on Pop!_OS is a general thing, could you maybe create an issue with reproduction steps so we can confirm this is the case and then potentially fix this? Thanks.
Tested with a fresh install of prisma-client @ 3.7.0, and the error did not occur anymore.