prisma: Timed out fetching a new connection from the pool
Bug description
After running my application for around 1-2 days, I start getting the following error:
prisma:client Error: Timed out fetching a new connection from the pool. Please consider reducing the number of requests or increasing the `connection_limit` parameter (https://www.prisma.io/docs/concepts/components/prisma-client/connection-management#connection-pool). Current limit: 10.
The same happened before with a connection limit of 3. Raising it to 10 seemed to have no effect.
After the error occurs once, no query will succeed anymore, the error will be thrown on any query and the application requires a restart (prisma will never recover from the error).
I checked whether the database has an issue, but it is working fine.
SHOW PROCESSLIST shows that no connection is actually established.
How to reproduce
I have not found a way to reliably reproduce this yet. I always need to wait 1-2 days until the error randomly appears.
I am running my application in debug mode with DEBUG=* for the last days, so if you need any additional information, I can look it up in the logs.
Expected behavior
No response
Prisma information
generator client {
provider = "prisma-client-js"
binaryTargets = ["native"]
}
datasource db {
provider = "mysql"
url = env("DATABASE_URL")
}
I can’t post my complete prisma schema here. It doesn’t matter which queries are run, all of them fail with the same error.
Environment & setup
- OS: Linux (official node:14 docker image)
- Database: MySQL 8.0
- Node.js version: 14
Prisma Version
prisma : 2.21.2
@prisma/client : 2.21.2
Current platform : darwin
Query Engine : query-engine e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/query-engine-darwin)
Migration Engine : migration-engine-cli e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/migration-engine-darwin)
Introspection Engine : introspection-core e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/introspection-engine-darwin)
Format Binary : prisma-fmt e421996c87d5f3c8f7eeadd502d4ad402c89464d (at node_modules/prisma/node_modules/@prisma/engines/prisma-fmt-darwin)
Default Engines Hash : e421996c87d5f3c8f7eeadd502d4ad402c89464d
Studio : 0.371.0
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 25
- Comments: 30 (14 by maintainers)
Hello,
We are experiencing the same problem as described in this issue.
The application is used in production and there are not too much workloads but we can’t figure out :
We see that there’s something involving mobc but don’t know how to implement it.
The application is a backend API developed with NestJS.
Could someone give us a hint please ?
Thanks
A sample log of the application:
I’ve been experiencing this as well. We get a temporary spike in requests to our app servers that increases memory and cpu near 100% for a short time. Prisma then seems to get stuck in a state of every query failing to fetch a connection and never recovers, even as cpu and memory usage drop. Restarting the servers returns things to normal. Wondering if https://github.com/prisma/prisma/issues/9584 is related since both seem to involve the client connections getting stuck in particular failure states.
We actually have a reproduction now (essentially load spike or resource constraint with specific parameters) and are working on understanding the problem (non surprisingly something with the connection pool) and hopefully fixing it (no ETA - but if you are able to somehow reproduce this, we are looking for testers of our fixes of course).
Some related issues: https://github.com/prisma/prisma/issues/11868 https://github.com/prisma/prisma-private/issues/160 (private with reproduction)
We’ve been experiencing a similar issue (
v3.9.2); after a day or two, all connections are rejected with theTimed out fetching new connectionissue. The pool never recovers; a restart is necessary. Meanwhile, the actual underlying PSQL connections are all unused and available for queries.Our issue may more closely match #11868, since there is no noticeable resource spike leading to the connection issue. We do execute a lot of queries in parallel, which seems to match the described behavior in that ticket. We’ll try serializing the queries and seeing if that changes anything.
The problem occured again and after further digging into metrics and logs, I am quite sure that this is not an issue with prisma, but with my web server running out of memory.
Still have the same error we need to figure a workaround
As mentioned above we are working on fixing this.
In the meantime, here is a possible temporary workaround you can try: Prisma offers a Client method to “disconnect” the current Engine from the Client:
prisma.$disconnect(). By calling that method, we had good results in having Prisma Client internally “restart” the connection pool which allows you to avoid the restart of your application or server. In our testing it was as easy as implementing an endpoint that calls just this method, calling it manually, and then new queries in other requests recovered instantly or after a short time period. (We also automated calling the endpoint for tests over longer periods of times with equally good results.)We realize that this is not a long term solution for the problem of course, but hope it can work as a temporary workaround to unblock you. We’ll update this issue when we are closer to a real solution.
Any update on this issue?
We experiencing the same. We have 2 prod servers that connect to 1 DB server. Randomly one of the serves get
After getting the error, every further request fails. And the prismaClient does not seem to recover at any time.
The other server remains stable.
A restart of the affected server solves the issue directly. But it can not be the solution to restart the server each time.
Backups are hard… Hope you get that fixed some time somehow. Until then, have fun with Prisma.
I have not changed both values, so
connect_timeoutshould be 5 andpool_timeoutshould have a value of 10.I was wrong when I said that the issue has nothing to do with the backup job. The graph I posted only showed the last hour, even though the last 12 hours of logs were processed. Here is a correct graph counting the amount of “Timed out fetching…” logs from the last 12 hours.
The first spike is at 3am, then at 4am and 5 am (the hourly backup).
Even after load is back to normal at the database server, Prisma keeps timing out trying to fetch a connection. A possible reproduction might be to put the database server under load (cpu and memory using some stress tester) until a timeout occurs in the Prisma app and then remove the load from the database server and see if Prisma is still timing out.
As before, simply restarting the Prisma application resolved the problem.
The reason the server ran out of memory was again restic. I never had critical problems with prismas memory usage. I decided to upscale the machine resources to have more memory. This should hopefully get rid of the issue. Thank you for your help in trying to find a solution! 😃
Ok, so filling up memory while running queries would be a viable reproduction attempt here.