next.js: [Error 500] "Socket Hang Up" Randomly Occurring on any Routes in Production Mode

I had the same issue, with next@13.4.x + next-auth@4.x.x, deployed to ECS in a standalone mode docker image. The server crash randomly with Error: socket hang up even under super light load. But when running next dev locally, it was all normal.

I tried all different kinds of troubleshooting, including changing AWS health check settings, moving all API routes from app dir to pages dir, trying different next-auth versions, setting httpAgentOptions.keepAlive to false, switching docker image base from alpine to debian, etc. None of which fixed the issue.

With a local docker container and k6 for load testing, I was finally able to reproduce the error. And after trying all different combinations, I found out the error starts from next@13.3.5-canary.9, and there’s the diff link https://github.com/vercel/next.js/compare/v13.3.5-canary.8...v13.3.5-canary.9 if any of you are interested in digging more into it. And 13.3.4 is the last stable version prior to it.

TLDR: use next@13.3.4

Update 18 Aug

Please ignore my workaround above. I just found out that 13.3.4 is also broken, just happened that it didn’t throw the error but crashes the server without printing any logs.

The true fix I found so far is to run a x64 image, and the error is gone even under heavy load:

docker buildx build --platform=linux/amd64 -t app-name .

However, this is far from ideal, it takes way longer for my M chip MacBook to build, and I’ll have to migrate my AWS server architecture from ARM to x64 😢.

And one more thing, the current latest version 13.4.18 is still broken in the standalone mode due to env variables bug, see https://github.com/vercel/next.js/issues/53367, so i have to test with the last working version 13.4.12. While the PR https://github.com/vercel/next.js/pull/54203 to fix the env variable bug has been merged, it might take a while to be released on npm.

@ielaajezdev saw you mention you tried linux/amd64 but did work, can you share more detail? like how you build the image and how you trigger the error?

Update 19 Aug

Found this issue https://github.com/prisma/prisma/issues/19419. And i tried the workaround posted by @tigranbs, setting Prisma’s DB url with ?connection_limit=1 fixed the error for me, but obviously it’s not the fix to this bug.

I also tried, prisma@4.16.2 and prisma@5.1.1, and getting the error for both. But when I load test on a route that doesn’t fetch data with Prisma, the error is gone.

So for me, the causes of the error can be narrowed down to:

• Arm64 Linux
• next@13.4.x + prisma@4/prisma@5
• More than 2 concurrent requests

@piotrcichosz @0xadada @mthmcalixto @renanrodrigueszup did your case follow this pattern?

Update 19 Aug

Tried the latest next@13.4.19, still getting the error under the conditions mentioned above, but the error message is different this time:

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11457:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async invokeRequest (/app/node_modules/.pnpm/next@13.4.19_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
    at async invokeRender (/app/node_modules/.pnpm/next@13.4.19_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/router-server.js:254:29)
    at async handleRequest (/app/node_modules/.pnpm/next@13.4.19_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/router-server.js:447:24)
    at async requestHandler (/app/node_modules/.pnpm/next@13.4.19_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/router-server.js:464:13)
    at async Server.<anonymous> (/app/node_modules/.pnpm/next@13.4.19_react-dom@18.2.0_react@18.2.0/node_modules/next/dist/server/lib/start-server.js:117:13) {
  cause: Error: connect ECONNREFUSED 0.0.0.0:39679
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '0.0.0.0',
    port: 39679
  }
}

By running netstat -tulpn | grep LISTEN before the load test, the process on port 39679 is:

tcp        0      0 0.0.0.0:39679           0.0.0.0:*               LISTEN      19/next-render-work

I would like to add my experience with this issue

• we don’t use Prisma in our project
• we don’t use Next Auth
• but still, the error appeared multiple times on prod
• had to downgrade to 13.3.4 and haven’t seen the error since then (at least, in our case)

So, I don’t think the issue is related to Prisma or NextAuth

In case anyone is using Sentry, our issue turned out to be related to a bug with the @sentry/nextjs package. Bumping it up a version has fixed the issue on our end.

https://github.com/orgs/vercel/discussions/3248#discussioncomment-7851868

I have just tried v14.0.4-canary.47 and the issue persists. I also tried Node.js v18 and v20.

We are only using the App Router. We do not use Prisma or NextAuth. This is affecting builds hosted on Vercel.com (including production).

It takes a little while for the issue to pop up after deploying, but after a few RSC renders, it happens quite often (~15% of the time).

Wow… this is really broad. Personally, I have a deployed project running 13.4.9 on bare Windows Server 2022 Datacenter, and it hasn’t had any of those errors. Though, I did get the errors while developing it, but at the time I believe it was some one-time-thing.

Also, 4 days ago on another project, I was messing around with next.js source code (in node_modules directly) to optimize API compression mechanism and I always got the errors consistenly. The first request completed/downloaded its content successfully, but I noticed the request was still going. I closed the tab and opened a new one, and it looked like the request just hung. Any requests to the server completely not responsive.

In that experiment I don’t think the render worker process died, I believe it is something else. I will try to replicate it, but I don’t think I will make an issue out of it (because in next.js defence, I modified its source code). Though I hope I will come with a theory that could explain the errors.

This error sometimes happens in developer mode in version 13.4.12, when there is a lot of refresh it stops working and needs to start the terminal again.

I’m also getting the error using next@13.4.19 and prisma for app/api route.

same here on 13.4.12

Failed to proxy http://127.0.0.1:59062/tapi Error: socket hang up
    at connResetException (node:internal/errors:717:14)
    at Socket.socketOnEnd (node:_http_client:526:23)
    at Socket.emit (node:events:525:35)
    at Socket.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ECONNRESET'
}
Failed to proxy http://127.0.0.1:59062/tapi Error: socket hang up
    at connResetException (node:internal/errors:717:14)
    at Socket.socketOnEnd (node:_http_client:526:23)
    at Socket.emit (node:events:525:35)
    at Socket.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ECONNRESET'
}
Failed to proxy http://127.0.0.1:59062/tapi Error: socket hang up
    at connResetException (node:internal/errors:717:14)
    at Socket.socketOnEnd (node:_http_client:526:23)
    at Socket.emit (node:events:525:35)
    at Socket.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ECONNRESET'
}

Just happened with me today using version 13.4.10. This app I’m worrking on is being deployed within a docker container on a EKS. Funny thing is, that moments before the container started logging the “Socket Hang Up” thing, the Ephemeral storage was nearly full, up to 86% usage. Maybe it could be something related to cache ? I’m keeping an eye on it trying to find a pattern.

Glad to see this thread since I am also experiencing frequent (seemingly random) “socket hang up errors” and it was quite hard to debug the root cause. I think my use case is different so I will add my error scenario just in case it makes the problem exploration easier.

Setup

• NextJS version: 13.4.11
• Using the pages/* and pages/api/* directories. not using the new app router

I am developing my next app in a Docker compose composition. The next app runs in the node 18 alpine image as per the Docker examples provided. Other containers in the composition are a postgres db, Prisma (studio), cerbos and strapi. I am developing on MacOS (Mac with M2). I have not yet deployed my app to production and only use development mode currently.

Answers to your questions

• The socket hang up occurs seemingly random and when it happens, I get about 3 socket hang up/ECONNRESET errors in a row.
• After a socket hang up, no other requests are handled anymore: I need to restart the container again to restore functionality.
• I thought at first that the problem was connected to my file API endpoint (an endpoint that downloads a file from a remote server and then sends it to the client after logging the request) but that does not seem to be the case: even when not loading any images, this error occurs.

Things I have tried

• I have tried to use the specific linux/amd64 platform for my Dockerfile and that did seem to resolve issues but was too slow in the development environment on my M2 mac
• I have tried to decrease the MTU for the docker bridge network as suggested here but that also did not resolve issues

Hope this helps! Following this issue and reverting to next 13.4.5 for now…

@SebastienSusini I also isolated this issue in my app to have started in next (same version in your bug report), and reverting to 13.4.5 resolved the issue13.4.6 🎉

13.4.12 with the same problem man

Same issue here, happening very often, impossible to find where it comes from.

I am using “next”: “^14.0.4”, with nextAuth, and nextJS middleware (my app uses also Wundergraph/sdk)

Any update on this issue?

Thx a lot

“next”: “^14.0.3” the same issue when running custom server

In our case it is not random. There is a feed.xml route in our app and it sends a request to S3 to get the actual feed. But the file size is about ~100mb, it gives “socket hang up” error and then it breaks the entire app. No way to handle the error in try-catch.

I’ve tried almost everything to get rid of this error, including trying to use node https or other 3rd party http clients instead of fetch api. But it seems, the problem is not related to next, it’s a node.js or maybe undici related problem. Also tried some other node images but I think a stable node image could fix this issue.

We could “solve” or at least work around the socket hang up errors by replacing node with bun.

After updating to Prisma 5.3.0, the issue seems to be resolved (in development) so it might be that the issues are unrelated, but I will continue to monitor.

Currently I have this issue very often (almost everyday). Situation is always the same: socket hang up, next-render-worker-app is missing in processes, server doesn’t respond - just hangs.

BTW prisma maintainers said they fixed related problem: https://github.com/prisma/prisma/issues/19419 – maybe this helps. I haven’t tried it yet since the fix is in unstable branch for now.

So for me, the causes of the error can be narrowed down to:

• Arm64 Linux
• next@13.4.x + prisma@4/prisma@5
• More than 2 concurrent requests

@piotrcichosz @0xadada @mthmcalixto @renanrodrigueszup did your case follow this pattern?

@mdluo not exactly, we don’t use prisma, and can see it occur after a single (non-concurrent) request.

@0xadada Interesting… It seems like your issue is a bit different but I think it correlates. I think this could be just the client aborting the connection and next.js still actively waiting for incoming TCP packets.

I’ll try to reproduce it by making a long request and abort it. See if I can get a reproduction.

We are also running into this issue, with the same circumstances as described before:

• Only seems to occur in production. (Possibly due to load?)
• Occurs only after a couple of days or a week+, after millions of requests have already happened
• Unable to pin-point one specific action/request that causes this issue

Error: socket hang up
    at connResetException (node:internal/errors:705:14)
    at Socket.socketOnEnd (node:_http_client:518:23)
    at Socket.emit (node:events:525:35)
    at Socket.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'ECONNRESET'

We will now downgrade to 13.4.5 and watch if it happens again, unfortunately we have no way to test it consistently. Time will tell…?

For info, we are running on AWS EC2 instances. @0xadada do I understand correctly that in your case following requests do get handled? For us it seems to completely stop the server from being able to handle any requests after that point.