vercel: Fetch failed with UND_ERR_CONNECT_TIMEOUT error on Next.js serverless function on Vercel Production env
TLDR; When executing a fetch request from a serverless function it sometimes fails returning a UND_ERR_CONNECT_TIMEOUT on a Nex.js production environment hosted on Vercel.
I currently have a Next.js site (v14.1.4 - Pages router) running on the Vercel platform (Node v20x) that performs fetch requests to the Slack API from an API route through a serverless function. This is a breakdown of what happens before the error:
- From a client-side component, start a fecth POST request to an API endpoint (route handler) on form submission;
- On an API serverless function realise another fetch POST request to an external API (in my case I used Slack message API - “https://slack.com/api/chat.postMessage”);
- On a production environment hosted on Vercel, around 70% of the requests to Slack are working while another 30% fail returning 500 server error code “UND_ERR_CONNECT_TIMEOUT”.
Full error message:
Unhandled Rejection: TypeError: fetch failed at node:internal/deps/undici/undici:12345:11 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { cause: ConnectTimeoutError: Connect Timeout Error at onConnectTimeout (node:internal/deps/undici/undici:7492:28) at node:internal/deps/undici/undici:7448:50 at Immediate._onImmediate (node:internal/deps/undici/undici:7480:13) at process.processImmediate (node:internal/timers:478:21) at process.callbackTrampoline (node:internal/async_hooks:130:17) { code: 'UND_ERR_CONNECT_TIMEOUT' } } Node.js process exited with exit status: 128. The logs above can help with debugging the issue.
IMPORTANT NOTE: I’ve seen this issue happening in both pages and app routers but more frequently on the pages router. Also, it only happens on production environments hosted on Vercel. I could not (personally) reproduce from my local server nor dev environments.
Reproducible By
This codepen contains an example of the code being used to trigger the fetch requests coming from the input form submission to the serverless function API call. https://codesandbox.io/p/devbox/gifted-shirley-mzlvgy?file=%2Fapp%2Fapi%2Froute.js%3A1%2C1-39%2C1
Expected Behaviour
Currently some external API calls from a serverless function are returning undici unhandled fetch errors. Expected behaviour is no errors being returned and API call succeeding every time.
Environment
Operating System: Vercel Servers
Binaries: Node: v20x (default vercel v20 setting) npm: 10.2.3 yarn: 1.22.19 build command: yarn build
Relevant Packages: next: 14.1.4 react: 18.2.0 react-dom: 18.2.0
Additional Context
Additional information about the issue and more cases can be found at: https://github.com/vercel/next.js/discussions/57384 https://github.com/vercel/next.js/issues/66373
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 18
- Comments: 96 (6 by maintainers)
Commits related to this issue
- fix(frontend): SSR req close connenction direclty https://github.com/vercel/vercel/issues/11692#issuecomment-2171172148 — committed to Aylie-Chou/congress-dashboard-monorepo by deleted user 2 years ago
- Use axios instead of fetch https://github.com/vercel/vercel/issues/11692#issuecomment-2664739637 — committed to ben-z/watcloud-website by ben-z a year ago
- Add concurrency limit to processImage and switch from fetch to axios (#4382) ## Description I'm using [Jules](https://jules.google) to automate some tasks in the website repo, but the tool can only u... — committed to WATonomous/watcloud-website by ben-z a year ago
Hi, I wanted to share an update on this discussion since a lot of people are impacted. Sorry for the delay that this is taking, it’s unfortunately a very complex issue that is also quite hard to debug, and seems to heavily depend on the code that is running (when CPU usage is high).
We’ve been working with @Uzlopak (Node.js and Undici contributor) to try to fix the issue, namely:
In parallel, we also added an option to transparently update the Undici version used by Node.js, without needing to manually install Undici on your projects. We tested all of those patches internally (as we’re also hitting this issue on production on some of our apps) and it reduced the frequency of
UND_ERR_CONNECT_TIMEOUTerrors by a lot.You can enable this option on your own projects by adding the following environment variable to your project, then redeploying:
VERCEL_UNDICI1Note that this means your Undici version will use this commit instead of the one bundled by Node.js -
process.versions.undiciwill show you the current and updated Undici version.Please let us know if this improves the situation on your side too 🙏, and I’m again sorry for the delay.
Hi,
I understand this is probably a very complex issue, and not something easy to fix. However, I have to say I am quite surprised about Vercel’s behavior on this issue.
This is not a small error that you can easily get around to. This is effectively bricking any app which uses any api endpoint. I would expect this to effectively treated as a burning fire, and have daily updates about how debugging is going and how the issue can be mitigated, potentially rolling back to a previous version where this issue didn’t arise.
Instead, this issue has been opened for more than 2 months. The last update we got was on June 16th, almost 2 months ago.
Could you help me understand what is going on? Does your monitoring indicate that this issue is very rare / affects only a small percentage of users? Because I cannot imagine that you know the entire functions API is broken but the issue is just ignored and swept under the rug.
This is serious enough to force people off your entire platform, even if they love it, as they have no choice - either that or just close shop for 6 months (or however long it will take for this to be fixed).
IMHO, you should give a convincing explanation as to why this has been ignored for so long, a clear path to recovery, and daily updates until resolution. Otherwise, I don’t see how one can take this platform seriously if things can just completely break for months with no apparent interest from Vercel.
Hi there. 👋🏻 We’ve been looking deeply into these issues within Vercel for the past few weeks and will continue to do so until we find the cause here. This is affecting a few internal projects, so we also feel your pain.
Pinpointing the exact issue is hard, and we have a few working theories as to what might be at fault here. We’ll keep you posted as we learn more, and remediate this issue.
We had to ditch our app from Vercel while this issue is still happening so often. Having API calls failing due to server instability is just not reliable enough for our business.
Some updates:
fetch()method. You can add aconnection: 'close'header to yourfetch()calls to disable keep-alive, which should help mitigating this issueHi,
I’m writing to follow up on the ongoing issue with fetching RSS feeds in my Vercel-deployed Next.js application. Despite implementing numerous suggested solutions and workarounds, the problem persists. Below is a detailed account of the troubleshooting steps I’ve taken:
Additional Notes:
Despite all these efforts, I still cannot reliably fetch RSS feeds in the production environment. The errors consistently point to connection timeouts or failures to establish a connection.
Given my extensive troubleshooting, this issue might be related to some aspect of the Vercel serverless environment that’s not immediately apparent. I would greatly appreciate any additional insights, advanced debugging techniques, or alternative approaches you could suggest.
Thank you for your continued support in resolving this critical issue.
@QuiiBz I can confirm that after deploying our app with the
VERCEL_UNDICI=1flag, we haven’t seen a singleUND_ERR_CONNECT_TIMEOUTerror in the last 24 hours! 🎉Here’s a chart showing the
UND_ERR_CONNECT_TIMEOUTerrors in the last 15 days:Thank you all for the hard work 🙏
I’m having the same issue, only on prod, and only sporadically. Is there a fix?
@gudmundur Do you have an ETA on this? We tried exponential backoff on the functions that we could, but some of our functions are called by 3rd party webhooks, and they only fire once. If your solution won’t be for another month or two, then we’ll be hosting on a separate EC2 instance to handle certain functions.
@craigRSA the error you’re seeing is probably because your client still held onto a socket that the server had already closed and then tried to re-use it. This usually happens when the client-side keep-alive timeout is higher than the server-side socket timeout. You can usually fix this by lowering
globalAgent.options.timeout(or better create a separate custom agent with lowertimeoutvalue and pass that specifically for requests to the host that causes the error).If you also control the server, you could also increase the server-side socket timeout instead.
Closing in favor of https://github.com/nodejs/undici/pull/3495 and https://github.com/nodejs/undici/issues/3410. We try to use this repo to discuss issues with the
vercelcli and related open source libraries.I’m experiencing this same issue on a brand new app that I imported into Vercel this morning. It fails to make a fetch to the QStash API (which is authenticated) in order to push a task. There was about 30 minutes where it started working. But outside of that, has not worked. The fetch is being made from a server component.
Next 14.2.3 Node 20.x
Hi, we’re looking at the issue but don’t have any updates just yet. Any minimal reproduction that doesn’t depend on hitting an API behind authentication could help a lot 🙏
https://github.com/nodejs/node/issues/53186 is very likely unrelated to this issue, since it’s only about Node.js 20 while this current discussion is affecting both Node.js 18 and Node.js 20.
I am currently trying to reach some nodejs maintainers to update undici in node v20 and v22, so that it is part of the next minors.
https://github.com/nodejs/node/pull/55503#issuecomment-2461858871
@gudmundur any updates on this? 🙏
I am also getting this issue and have tried to go through each proposed solution above. Still can’t get to a predictable/ reliable state.
I guess we’re just waiting on a fix for now.
I’m getting more and more of these errors. Also using axios for requests. Can someone at vercel please prioritize this?
On Wed, 19 Jun 2024 at 22:09, Abhi Aiyer @.***> wrote:
I successfully resolved the issue by configuring the undici global dispatcher in the root layout.
Yup, in project settings -> Environment Variables
@andremendonca03 I was able to resolve my problem by setting
--no-experimental-fetchinNODE_OPTIONSenvironment variable in vercel. It also helped me to get better error log which required me to increase timeout innext.config.jsasstaticPageGenerationTimeout: 1000,Seems I’m encountering the same thing - no issues locally - timeouts on Vercel.
Running next 14.2.3
I am also getting this error but for my project it is on build time. And it only happens in vercel, no issues in local. The API times are also not unreasonable i.e. 2 seconds per API call at max. (response time is local). Not sure how I can measure in vercel at build time.
Nextjs version is 13.1.6 and I’ve tried Node 18x and 20x + node_options
Hi, what version of node has 6.21?
There’s no current patch of node on 6.21. Downgrade to Node 18. Vercel does not let you run semver node installs regardless, you are limited to major version so this has to be fixed within Vercel.
Overnight we started seeing “TypeError: Response.clone: Body has already been consumed.”. All projects on Next 14.1.4 are unable to deploy. This looks like a configuration change in Vercel which has prompted deployment crashes.
My best guess was that it’s related to the fact that the global http & https agents have keep-alive enabled by default in Node.js 18 or later (I assume that’s what the linked fixes are related to). Actually managed to make it happen locally as well when I bursted out enough requests. Disabling keep-alive fixed it for now but will look into finding a better timeout value later on.
Edit: can confirm that setting
VERCEL_UNDICI=1in my dot-env file also fixed it locally (haven’t tried deploying yet).@Uzlopak It is not public and I went through the code to check to see if any awaits were missed and could not find any. Checking the overnight logs and with the VERCEL_UNDICI=1 flag removed we still no longer have any TypeError: Response.clone: Body has already been consumed.
I then readded the flag this morning and immediately ran into this
Unsure if this is related to the flag or something else in undici but the only thing that I have changed is that environment variable
@ngroenewold95
I actually wonder how you could get an Unhandled Rejection Error. Are you not awaiting the fetch?
I am now investigating for like 2-3 hours to figure out how you could get that unhandled rejection issue. Do you have your code public viewable?
here’s the code which worked for us
I faced this issue while i was working on a slack slash commands API, vercel provides a waitUntil feature which immediately close the response but the waitUntil request can proceed.
https://vercel.com/docs/functions/functions-api-reference#waituntil
You probably want to use
waitUntilto benefit from the best of both worlds: https://vercel.com/docs/functions/functions-api-reference#waituntilIndeed, some webhook providers expect a response within a limited number of seconds which could be problematic if your processing lasts longer than that duration.
Is there any fix for this issue or end up dropping Vercel.
either going to move all API endpoints off of vercel directly, or the entire project at this point
Hi!
If you don’t plan to provide an ETA for the resolution, does it mean that we can consider this issue abandoned and just try to move our hosting somewhere else? I have an API route which is completely useless at the moment - thankfully we’re not launched yet, but having this happen while real traffic was coming would have been a complete disaster.
My application is using
axiosto send requests. To my knowledge, axios does not send a keepAlive so I’m not sure the recommendation helps. In our vercel logs we are seeing:We are on Node 18.
So this issue comes and goes for us… It started up again last week, which seems to match this thread. Here is a comparison of the number of UND_ERR_CONNECT_TIMEOUT’s we’ve gotten vs the general activity of our platform.
This is in production, on Vercel. These seem to mostly be internal calls, like GET /api/auth/session as well as webhooks.
I’m getting a build error: headers not found
On Mon, 10 Jun 2024, 20:29 Pratik Kataria, @.***> wrote:
I got a response from Vercel support, here’s an excerpt:
They are basically looping back into this thread 😃 I tried both flags:
--no-experimental-fetchand--dns-result-order=ipv4firstbut with the first the build fails and second doesn’t seem to do anything.@andremendonca03 using internal API, no third party API. Our API is GraphQL but it used to work without any issue in past on vercel.
Which external APIs are you trying to fetch specifically and which method are you using? @zackproser @kpratik2015