vercel: Fetch failed with UND_ERR_CONNECT_TIMEOUT error on Next.js serverless function on Vercel Production env

TLDR; When executing a fetch request from a serverless function it sometimes fails returning a UND_ERR_CONNECT_TIMEOUT on a Nex.js production environment hosted on Vercel.

I currently have a Next.js site (v14.1.4 - Pages router) running on the Vercel platform (Node v20x) that performs fetch requests to the Slack API from an API route through a serverless function. This is a breakdown of what happens before the error:

  1. From a client-side component, start a fecth POST request to an API endpoint (route handler) on form submission;
  2. On an API serverless function realise another fetch POST request to an external API (in my case I used Slack message API - “https://slack.com/api/chat.postMessage”);
  3. On a production environment hosted on Vercel, around 70% of the requests to Slack are working while another 30% fail returning 500 server error code “UND_ERR_CONNECT_TIMEOUT”.

Full error message: Unhandled Rejection: TypeError: fetch failed at node:internal/deps/undici/undici:12345:11 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { cause: ConnectTimeoutError: Connect Timeout Error at onConnectTimeout (node:internal/deps/undici/undici:7492:28) at node:internal/deps/undici/undici:7448:50 at Immediate._onImmediate (node:internal/deps/undici/undici:7480:13) at process.processImmediate (node:internal/timers:478:21) at process.callbackTrampoline (node:internal/async_hooks:130:17) { code: 'UND_ERR_CONNECT_TIMEOUT' } } Node.js process exited with exit status: 128. The logs above can help with debugging the issue.

IMPORTANT NOTE: I’ve seen this issue happening in both pages and app routers but more frequently on the pages router. Also, it only happens on production environments hosted on Vercel. I could not (personally) reproduce from my local server nor dev environments.

Reproducible By

This codepen contains an example of the code being used to trigger the fetch requests coming from the input form submission to the serverless function API call. https://codesandbox.io/p/devbox/gifted-shirley-mzlvgy?file=%2Fapp%2Fapi%2Froute.js%3A1%2C1-39%2C1

Expected Behaviour

Currently some external API calls from a serverless function are returning undici unhandled fetch errors. Expected behaviour is no errors being returned and API call succeeding every time.

Environment

Operating System: Vercel Servers

Binaries: Node: v20x (default vercel v20 setting) npm: 10.2.3 yarn: 1.22.19 build command: yarn build

Relevant Packages: next: 14.1.4 react: 18.2.0 react-dom: 18.2.0

Additional Context

Additional information about the issue and more cases can be found at: https://github.com/vercel/next.js/discussions/57384 https://github.com/vercel/next.js/issues/66373

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 18
  • Comments: 96 (6 by maintainers)

Commits related to this issue

Most upvoted comments

Hi, I wanted to share an update on this discussion since a lot of people are impacted. Sorry for the delay that this is taking, it’s unfortunately a very complex issue that is also quite hard to debug, and seems to heavily depend on the code that is running (when CPU usage is high).

We’ve been working with @Uzlopak (Node.js and Undici contributor) to try to fix the issue, namely:

In parallel, we also added an option to transparently update the Undici version used by Node.js, without needing to manually install Undici on your projects. We tested all of those patches internally (as we’re also hitting this issue on production on some of our apps) and it reduced the frequency of UND_ERR_CONNECT_TIMEOUT errors by a lot.

You can enable this option on your own projects by adding the following environment variable to your project, then redeploying:

  • Key: VERCEL_UNDICI
  • Value: 1

Note that this means your Undici version will use this commit instead of the one bundled by Node.js - process.versions.undici will show you the current and updated Undici version.

Please let us know if this improves the situation on your side too 🙏, and I’m again sorry for the delay.

Hi,

I understand this is probably a very complex issue, and not something easy to fix. However, I have to say I am quite surprised about Vercel’s behavior on this issue.

This is not a small error that you can easily get around to. This is effectively bricking any app which uses any api endpoint. I would expect this to effectively treated as a burning fire, and have daily updates about how debugging is going and how the issue can be mitigated, potentially rolling back to a previous version where this issue didn’t arise.

Instead, this issue has been opened for more than 2 months. The last update we got was on June 16th, almost 2 months ago.

Could you help me understand what is going on? Does your monitoring indicate that this issue is very rare / affects only a small percentage of users? Because I cannot imagine that you know the entire functions API is broken but the issue is just ignored and swept under the rug.

This is serious enough to force people off your entire platform, even if they love it, as they have no choice - either that or just close shop for 6 months (or however long it will take for this to be fixed).

IMHO, you should give a convincing explanation as to why this has been ignored for so long, a clear path to recovery, and daily updates until resolution. Otherwise, I don’t see how one can take this platform seriously if things can just completely break for months with no apparent interest from Vercel.

Hi there. 👋🏻 We’ve been looking deeply into these issues within Vercel for the past few weeks and will continue to do so until we find the cause here. This is affecting a few internal projects, so we also feel your pain.

Pinpointing the exact issue is hard, and we have a few working theories as to what might be at fault here. We’ll keep you posted as we learn more, and remediate this issue.

We had to ditch our app from Vercel while this issue is still happening so often. Having API calls failing due to server instability is just not reliable enough for our business.

Some updates:

  • We landed a network improvement for builds to mitigate timeout errors at build time
  • We believe that the issue at runtime is caused by a race condition regarding keep-alive settings in Undici, the library used by Node.js to provide the fetch() method. You can add a connection: 'close' header to your fetch() calls to disable keep-alive, which should help mitigating this issue

Hi,

I’m writing to follow up on the ongoing issue with fetching RSS feeds in my Vercel-deployed Next.js application. Despite implementing numerous suggested solutions and workarounds, the problem persists. Below is a detailed account of the troubleshooting steps I’ve taken:

  1. Original Implementation:
  • Used axios to fetch the RSS feed
  • Encountered ETIMEDOUT errors
  1. DNS Lookup Test:
  • Implemented DNS lookup before the fetch request
  • Result: DNS resolution successful, but connection issues persisted
  1. Alternative HTTP Clients:
  • Tested with node-fetch, native https module, and axios with custom agent
  • Implemented in separate API routes: /api/api-node-fetch, /api/api-https, /api/api-axios
  • Result: All methods encountered similar timeout issues
  1. Circuit Breaker Pattern:
  • Implemented a circuit breaker using axios
  • API route: /api/api-circuit-breaker
  • Result: Did not resolve the timeout issues
  1. HTTP Instead of HTTPS:
  • Tested fetching over HTTP instead of HTTPS
  • API route: /api/api-http
  • Result: Still encountered connection issues
  1. Undici Implementation:
  • Used the undici package as suggested
  • Implemented in /api/api-undici
  • Added User-Agent header
  • Result: Connection issues persisted
  1. Undici with Custom Agent:
  • Implemented a custom Undici agent with a specific configuration
  • Used AbortController for timeout handling
  • API route: /api/api-undici-custom
  • Result: Did not resolve the issue
  1. Connection: Close Header:
  • Implemented the most recent suggestion to add ‘Connection’: ‘close’ header
  • Used Undici with this header to disable keep-alive
  • API route: /api/api-undici-close
  • Result: The issue still persists

Additional Notes:

  • All these methods work correctly in the local development environment
  • The issue occurs only in the Vercel production environment
  • The problem is consistent across multiple RSS feed sources, not limited to a single domain

Despite all these efforts, I still cannot reliably fetch RSS feeds in the production environment. The errors consistently point to connection timeouts or failures to establish a connection.

Given my extensive troubleshooting, this issue might be related to some aspect of the Vercel serverless environment that’s not immediately apparent. I would greatly appreciate any additional insights, advanced debugging techniques, or alternative approaches you could suggest.

Thank you for your continued support in resolving this critical issue.

Screenshot 2024-06-22 at 12 47 01

@QuiiBz I can confirm that after deploying our app with the VERCEL_UNDICI=1 flag, we haven’t seen a single UND_ERR_CONNECT_TIMEOUT error in the last 24 hours! 🎉

Here’s a chart showing the UND_ERR_CONNECT_TIMEOUT errors in the last 15 days: image

Thank you all for the hard work 🙏

I’m having the same issue, only on prod, and only sporadically. Is there a fix?

@gudmundur Do you have an ETA on this? We tried exponential backoff on the functions that we could, but some of our functions are called by 3rd party webhooks, and they only fire once. If your solution won’t be for another month or two, then we’ll be hosting on a separate EC2 instance to handle certain functions.

@craigRSA the error you’re seeing is probably because your client still held onto a socket that the server had already closed and then tried to re-use it. This usually happens when the client-side keep-alive timeout is higher than the server-side socket timeout. You can usually fix this by lowering globalAgent.options.timeout (or better create a separate custom agent with lower timeout value and pass that specifically for requests to the host that causes the error).

If you also control the server, you could also increase the server-side socket timeout instead.

Closing in favor of https://github.com/nodejs/undici/pull/3495 and https://github.com/nodejs/undici/issues/3410. We try to use this repo to discuss issues with the vercel cli and related open source libraries.

I’m experiencing this same issue on a brand new app that I imported into Vercel this morning. It fails to make a fetch to the QStash API (which is authenticated) in order to push a task. There was about 30 minutes where it started working. But outside of that, has not worked. The fetch is being made from a server component.

Next 14.2.3 Node 20.x

Hi, we’re looking at the issue but don’t have any updates just yet. Any minimal reproduction that doesn’t depend on hitting an API behind authentication could help a lot 🙏

https://github.com/nodejs/node/issues/53186 is very likely unrelated to this issue, since it’s only about Node.js 20 while this current discussion is affecting both Node.js 18 and Node.js 20.

I am currently trying to reach some nodejs maintainers to update undici in node v20 and v22, so that it is part of the next minors.

https://github.com/nodejs/node/pull/55503#issuecomment-2461858871

@gudmundur any updates on this? 🙏

I am also getting this issue and have tried to go through each proposed solution above. Still can’t get to a predictable/ reliable state.

I guess we’re just waiting on a fix for now.

I’m getting more and more of these errors. Also using axios for requests. Can someone at vercel please prioritize this?

On Wed, 19 Jun 2024 at 22:09, Abhi Aiyer @.***> wrote:

My application is using axios to send requests. To my knowledge, axios does not send a keepAlive so I’m not sure the recommendation helps. In our vercel logs we are seeing:

We are on Node 18.

Unhandled Rejection: TypeError: fetch failed at node:internal/deps/undici/undici:12618:11 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async h.send (/var/task/apps/frontend/.next/server/chunks/1542.js:1:73006) { cause: ConnectTimeoutError: Connect Timeout Error at onConnectTimeout (node:internal/deps/undici/undici:7760:28) at node:internal/deps/undici/undici:7716:50 at Immediate._onImmediate (node:internal/deps/undici/undici:7748:13) at process.processImmediate (node:internal/timers:476:21) at process.topLevelDomainCallback (node:domain:161:15) at process.callbackTrampoline (node:internal/async_hooks:126:24) { code: ‘UND_ERR_CONNECT_TIMEOUT’ } } Node.js process exited with exit status: 128. The logs above can help with debugging the issue.

— Reply to this email directly, view it on GitHub https://github.com/vercel/vercel/issues/11692#issuecomment-2179458591, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM44FMY4O5IDWCJY2RLTKDZIHXQDAVCNFSM6AAAAABIY4K6QCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZZGQ2TQNJZGE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

I successfully resolved the issue by configuring the undici global dispatcher in the root layout. CleanShot 2024-06-17 at 20 03 50@2x

@andremendonca03 I was able to resolve my problem by setting --no-experimental-fetch in NODE_OPTIONS environment variable in vercel. It also helped me to get better error log which required me to increase timeout in next.config.js as staticPageGenerationTimeout: 1000,

@kpratik2015 is you set --no-experimental-fetch do you also need to import node-fetch?

No, I’ve made no code changes (apart from config change I mentioned) and we continue to simply use fetch with Node v20 set in Vercel.

@PratikKataria-plivo how do you set the env variable on vercel? Is it just a new variable name NODE_OPTIONS and value --no-experimental-fetch?

Yup, in project settings -> Environment Variables

image

@andremendonca03 I was able to resolve my problem by setting --no-experimental-fetch in NODE_OPTIONS environment variable in vercel. It also helped me to get better error log which required me to increase timeout in next.config.js as staticPageGenerationTimeout: 1000,

Seems I’m encountering the same thing - no issues locally - timeouts on Vercel.

Running next 14.2.3

I am also getting this error but for my project it is on build time. And it only happens in vercel, no issues in local. The API times are also not unreasonable i.e. 2 seconds per API call at max. (response time is local). Not sure how I can measure in vercel at build time. image

Nextjs version is 13.1.6 and I’ve tried Node 18x and 20x + node_options image

Hi, what version of node has 6.21?

Solution is to downgrade to Node 18. the issue is directly within Node 20.18, Undici 6.19. Fixed in Undici 6.20 so we’ll have to wait for another node patch

I’ve checked the descriptions I don’t know what version of the node is using the Undici 6.21 version

May I know how to check which version of the node is using undici 6.21 version?

There’s no current patch of node on 6.21. Downgrade to Node 18. Vercel does not let you run semver node installs regardless, you are limited to major version so this has to be fixed within Vercel.

Overnight we started seeing “TypeError: Response.clone: Body has already been consumed.”. All projects on Next 14.1.4 are unable to deploy. This looks like a configuration change in Vercel which has prompted deployment crashes.

My best guess was that it’s related to the fact that the global http & https agents have keep-alive enabled by default in Node.js 18 or later (I assume that’s what the linked fixes are related to). Actually managed to make it happen locally as well when I bursted out enough requests. Disabling keep-alive fixed it for now but will look into finding a better timeout value later on.

import { globalAgent } from 'https'

globalAgent.options.keepAlive = false
// or
globalAgent.options.timeout = 60_000

Edit: can confirm that setting VERCEL_UNDICI=1 in my dot-env file also fixed it locally (haven’t tried deploying yet).

@Uzlopak It is not public and I went through the code to check to see if any awaits were missed and could not find any. Checking the overnight logs and with the VERCEL_UNDICI=1 flag removed we still no longer have any TypeError: Response.clone: Body has already been consumed.

I then readded the flag this morning and immediately ran into this

image

Unsure if this is related to the flag or something else in undici but the only thing that I have changed is that environment variable

@ngroenewold95

I actually wonder how you could get an Unhandled Rejection Error. Are you not awaiting the fetch?

I am now investigating for like 2-3 hours to figure out how you could get that unhandled rejection issue. Do you have your code public viewable?

here’s the code which worked for us image

I faced this issue while i was working on a slack slash commands API, vercel provides a waitUntil feature which immediately close the response but the waitUntil request can proceed.

https://vercel.com/docs/functions/functions-api-reference#waituntil

I spent a day debugging my problem thinking it was related to this issue. Even though it’s not the exact thing that is being discussed here I’ll post the solution to my problem in case anyone else stumbles upon it.

In my case I was using a webhook that I copied from a guide somewhere. It always worked just fine on localhost

// Non-blocking call to process the webhook event.
void processWebhookEvent(rawData);

return new Response("OK", { status: 200 });

Turns out this does not work in production as the function probably shuts down when Response is returned. I just used await to fix it:

await processWebhookEvent(rawData);

return new Response("OK", { status: 200 });

You probably want to use waitUntil to benefit from the best of both worlds: https://vercel.com/docs/functions/functions-api-reference#waituntil

Indeed, some webhook providers expect a response within a limited number of seconds which could be problematic if your processing lasts longer than that duration.

Is there any fix for this issue or end up dropping Vercel.

either going to move all API endpoints off of vercel directly, or the entire project at this point

Hi!

If you don’t plan to provide an ETA for the resolution, does it mean that we can consider this issue abandoned and just try to move our hosting somewhere else? I have an API route which is completely useless at the moment - thankfully we’re not launched yet, but having this happen while real traffic was coming would have been a complete disaster.

My application is using axios to send requests. To my knowledge, axios does not send a keepAlive so I’m not sure the recommendation helps. In our vercel logs we are seeing:

We are on Node 18.

Unhandled Rejection: TypeError: fetch failed
at node:internal/deps/undici/undici:12618:11
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async h.send (/var/task/apps/frontend/.next/server/chunks/1542.js:1:73006) {
cause: ConnectTimeoutError: Connect Timeout Error
at onConnectTimeout (node:internal/deps/undici/undici:7760:28)
at node:internal/deps/undici/undici:7716:50
at Immediate._onImmediate (node:internal/deps/undici/undici:7748:13)
at process.processImmediate (node:internal/timers:476:21)
at process.topLevelDomainCallback (node:domain:161:15)
at process.callbackTrampoline (node:internal/async_hooks:126:24) {
code: 'UND_ERR_CONNECT_TIMEOUT'
}
}
Node.js process exited with exit status: 128. The logs above can help with debugging the issue.

So this issue comes and goes for us… It started up again last week, which seems to match this thread. Here is a comparison of the number of UND_ERR_CONNECT_TIMEOUT’s we’ve gotten vs the general activity of our platform.

CleanShot 2024-06-11 at 11 00 45@2x

This is in production, on Vercel. These seem to mostly be internal calls, like GET /api/auth/session as well as webhooks.

I’m getting a build error: headers not found

On Mon, 10 Jun 2024, 20:29 Pratik Kataria, @.***> wrote:

What build error are you guys getting with --no-experimental-fetch ? Because only after this flag I was able to get sensible error which pointed me to https://nextjs.org/docs/messages/page-data-collection-timeout which mentions: Increase the timeout by changing the config.staticPageGenerationTimeout configuration option (default 60 in seconds).

image.png (view on web) https://github.com/vercel/vercel/assets/14140930/a0622f5a-741e-44a1-b276-2e4becb94738

— Reply to this email directly, view it on GitHub https://github.com/vercel/vercel/issues/11692#issuecomment-2159136025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM44FNNPE6V3O5GNZWSJKLZGX5CVAVCNFSM6AAAAABIY4K6QCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJZGEZTMMBSGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

I got a response from Vercel support, here’s an excerpt:

Looking at your Runtime logs, I see both Edge Functions and Serverless Functions experience 504 issues.

Without going into details, I can confirm the two runtimes use different providers, so it’s unlikely to be a Vercel platform issue. I also couldn’t find similar reports from other customers, which would indicate that the issue may be with your backend.

Looking at your (redacted) Serverless Function, over the past 24 hours, there were over 1000 successful invocations and 94 timeouts.

Since the issue is intermittent and this function typically resolves within 1 second, you may by able to work around the issue by implementing a retry strategy where you abort and retry the POST request if a response from your backend hasn’t been received in 2 seconds.

You can also try to implement the different workarounds suggested in the Github issues below:

https://github.com/vercel/vercel/issues/11692#issuecomment-2152859828 https://github.com/vercel/next.js/issues/66373#issuecomment-2148546390

They are basically looping back into this thread 😃 I tried both flags: --no-experimental-fetch and --dns-result-order=ipv4first but with the first the build fails and second doesn’t seem to do anything.

Which external APIs are you trying to fetch specifically and which method are you using? @zackproser @kpratik2015

@andremendonca03 using internal API, no third party API. Our API is GraphQL but it used to work without any issue in past on vercel.

Which external APIs are you trying to fetch specifically and which method are you using? @zackproser @kpratik2015