firebase-functions: Node 8 - 'connection error'

Hi, after switching our functions to NODE 8 we are seeing “random” function failures with a message connection error. It is always right before resource id is replaced. There is no additional info provided.

(MSSQL pool create: null means we that connection was established without error)

[REQUIRED] Version info

node: 8.14.0 firebase-functions: 2.2.1 firebase-tools: 6.5.0 firebase-admin:

[REQUIRED] Test case

[REQUIRED] Steps to reproduce

[REQUIRED] Expected behavior

[REQUIRED] Actual behavior

‘connection error’ fail message with severity DEBUG without additional information

{
   "textPayload": "Function execution took 48 ms, finished with status: 'connection error'",
   "insertId": "000000-2739f8e3-4c81-4586-8931-a89020d19659",
   "resource": {
     "type": "cloud_function",
     "labels": {
       "project_id": " *** ",
       "region": "us-central1",
       "function_name": " *** "
     }
   },
   "timestamp": "2019-03-26T10:46:27.464788799Z",
   "severity": "DEBUG",
   "labels": {
     "execution_id": "vopi12sigsg0"
   },
   "logName": "projects/ *** /logs/cloudfunctions.googleapis.com%2Fcloud-functions",
   "trace": "projects/ *** /traces/40116ed33fcca88d605290469693fba0",
   "receiveTimestamp": "2019-03-26T10:46:32.776101066Z"
 },

Were you able to successfully deploy your functions?

yes

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 33 (9 by maintainers)

Links to this issue

Firebase Functions: Unclear "connection error" - Stack Overflow

Most upvoted comments

Yes @Neilpoulin I have the same issue that just showed up this week. My cloud function worked fine on node 8 until about 3 days ago when it started failing. I’m also using typescript, so I’m sure I’m returning promises correctly (i’ve also checked and double checked to be sure), and I’m only querying/updating 46 documents - so it should be an easy operation.

Finally after finding this thread - I was able to fix it by updating to node 10.

Side note: I was on node 10 about a month ago, but the console logs are so much harder to navigate (see this thread: https://twitter.com/JustinNoelDev/status/1207360381041156096) I guess for now I’ll keep it at node 10, and file a separate issue for that.

Update: fix for console.log objects showing up clean in node 10 here: https://github.com/firebase/firebase-functions/issues/612#issuecomment-646652675

corysmc on Aug 20, 2020

So, i just ran into this issue with a cloud function that processes ~10,000 records from firestore. I use promises (via async/await, transpiled via TypeScript) to process all of the records in a collection in batches of a configurable size. The function was regularly failing with the error code ‘connection error’, which, sadly, doesn’t show up as an error in any of the Firebase function logs or dashboards.

This same code is used in other situations/cloud functions with great success and has been working well for weeks (or maybe even months).

I’ve tried changing the batch size (from 500 to values as low as 100 and as high as 1000), adding a timeout between processing batches to ensure i wasn’t creating too many connections too quickly, all with no success. The function fails well before the timeout value and it still has plenty of memory. I have the timeout set to 540s and the memory at 1GB. Max memory useage appears to be just below 250MB.

Because this thread made it seemed related to the Node 8 runtime, I just tried changing my function to use the Node 10 (Beta) runtime. My function succeeded on the first try after making this runtime change.

Random Note: I noticed recently that the graphs in the Functions dashboard (https://console.cloud.google.com/functions/details/us-central1/xxxx) hadn’t been working until recently, right around the time I started having trouble with my functions failing with “connection error”.

neilpoulin on Jan 22, 2020

I have some updates! The GCF team has identified a workaround that prevents the connection error issue from happening, and they are in the process of rolling this out to production in the next week. I’ll update this thread when the fix is live. Thanks again for everyone’s patience!

thechenky on May 7, 2019

Hi @m4recek, @shaneosullivan I’m sorry you’re still struggling with this issue. I’ve checked on the internal bug and the GCF team is working hard to resolve this. I have pinged the bugs again to make sure your concerns have been passed on. Will post more updates as I have them. Apologies for this frustrating experience!

thechenky on Apr 29, 2019

Hi, is there any update?

Being on Node 6 is a bit of a challenge nowadays. Today we cant release because gaxios just dropped support for Node 6. Which is dependency of Google’s pubsub.

Current state is: Node 6 - will reach end-of-life in 4 days, more and more packages are dropping support Node 8 - has this bug for at least a month now Node 10 - is in beta

What is expected upgrade path? Is there any ETA for this to be resolved?

m4recek on Apr 26, 2019

Thanks for chiming in @shaneosullivan and thanks for the kind offer. Unfortunately this issue cannot be resolved in the public facing functions SDK (this repo) as it looks like a Google Cloud Functions infrastructure issue with the Node 8 runtime. I’ll check in on the internal bug we have to track this and post any updates here.

thechenky on Apr 3, 2019

Hi all! We are in the middle of rolling out the fix to production 😃 so glad to see its effects are already showing! I will update this thread when the rollout completes. Thanks everyone!

thechenky on May 15, 2019

Removing the no-recent-activity label so this doesn’t get automatically closed

joehan on May 7, 2019