firebase-functions: Firestore/triggers grpc-js No connection established errors

Related issues

Version info

node: 10.15.1

firebase-functions: 3.3.0

firebase-tools: 7.6.0

firebase-admin: 8.6.0

Test case

exports.addDriverLocation = functions.region(region).firestore
  .document('drivers/{userId}/locations/{locationId}')
  .onCreate((snap, context) => {
    const { userId = -3 } = context.params;
    const record = snap.data();
    const { location = {}} = record;
    const {
      uuid,
      coords: {
        latitude,
        longitude
      } = {},
      extras
    } = location;

    return snap.ref.set({
      userId,
      // server timestamp
      sts: admin.firestore.Timestamp.fromDate(new Date()),
      // client timestamp
      cts: admin.firestore.Timestamp.fromDate(new Date(timestamp)),
      latitude,
      longitude,
      extras
    });

Steps to reproduce

Create a firestore trigger function

Expected behavior

Every call should be get executed without gRPC related errors.

Actual behavior

Every now and then we get a No connection established error:

Error: No connection established
    at Http2CallStream.call.on (/srv/functions/node_modules/@grpc/grpc-js/build/src/client.js:96:45)
    at Http2CallStream.emit (events.js:194:15)
    at Http2CallStream.EventEmitter.emit (domain.js:459:23)
    at process.nextTick (/srv/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

It encountered 8600 exact errors for the past week:

Cloud Functions erors view

For every other firestore operation, we used old grpc as described here https://github.com/googleapis/nodejs-firestore/issues/768#issuecomment-537311719:

const Firestore = require('@google-cloud/firestore');
const grpc = require('grpc');
const firestore = new Firestore({ grpc });

but we don’t know how to set old grpc for functions.region(region).firestore to get rid of all these errors.

We are not sure if here is the right place to post this issue or the https://github.com/googleapis/nodejs-firestore. If this is the case, the please close this issue and we’ll go and report it on the nodejs-firestore repo.

Were you able to successfully deploy your functions?

No errors during functions deploy.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 5
Comments: 52

Most upvoted comments

I had a rogue reference in my package-lock.json

After nuking that and starting from a clean slate I was able to get the correct grpc-js version. For anyone else running into this issue right now, you need to delete BOTH your node_modules and your package-lock.json

For those interested it was under google-gax as a required dependency. It should read:

@grpc/grpc-js": "0.6.9"

I have since had three scheduled calls go off with no failures. Fingers crossed this is the one, thanks for the timely responses from all and the community support!

+10

mrmodelo on Oct 18, 2019

@adamkpurdy It seems to be sporadically failing on our project still.

jhk115 on Oct 18, 2019

Hey peoples:

I just did a dep upgrade after reading grpc/grpc-node#1027:

    "firebase-admin": "^8.6.0",
    "firebase-functions": "^3.3.0"

I’ve had three successful triggers fired. Anybody else wanna try and post their results?

adamkpurdy on Oct 18, 2019

Same error here. No solution yet?

othondev on Feb 5, 2021

This started popping up again today for me inside a cloud function.

Error: 14 UNAVAILABLE: No connection established
    at Object.callErrorFromStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/var/task/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/var/task/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/var/task/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at Http2CallStream.cancelWithStatus (/var/task/node_modules/@grpc/grpc-js/build/src/call-stream.js:457:14)
    at ChannelImplementation.tryPick (/var/task/node_modules/@grpc/grpc-js/build/src/channel.js:237:32)
    at Object.updateState (/var/task/node_modules/@grpc/grpc-js/build/src/channel.js:106:26)
Caused by: Error
    at WriteBatch.commit (/var/task/node_modules/@google-cloud/firestore/build/src/write-batch.js:415:23)
    at DocumentReference.update (/var/task/node_modules/@google-cloud/firestore/build/src/reference.js:378:14)
    at _callee9$ (/var/task/node_modules/elusive/dist/utils-f06f54c4.js:277:69)
    at tryCatch (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:61:40)
    at Generator.invoke [as _invoke] (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:290:22)
    at Generator.prototype.<computed> [as next] (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:113:21)
    at asyncGeneratorStep (/var/task/node_modules/elusive/dist/asyncToGenerator-7a28bf2e.js:752:24)
    at _next (/var/task/node_mo

samstr on Jun 23, 2020

I’m also getting the same issue using firebase-admin v8.10.0. Fails consistently, then works consistently a few times, then continues to fail.

chardos on May 31, 2020

This issue is not solved. Deleting package-lock.json and npm modules doesn’t solve the problem.

iustin-nita on Mar 18, 2020

Just for a record, I started to experience the same exact problem 3 days ago. Clean slate does not really help. Spontaneously some function will run but the majority will fail. The logs output snippet is this: "events.js:174

    throw er; // Unhandled 'error' event
    ^

Error: 14 UNAVAILABLE: No connection established at Object.callErrorFromStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call.js:31:26) at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client.js:327:49) at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181) at Http2CallStream.outputStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:117:74) at Http2CallStream.maybeOutputStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:156:22) at Http2CallStream.endCall (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:142:18) at Http2CallStream.cancelWithStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:467:14) at ChannelImplementation.tryPick (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/channel.js:238:32) at ChannelImplementation._startCallStream (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/channel.js:268:14) at Http2CallStream.start (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/call-stream.js:445:22) Emitted ‘error’ event at: at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client.js:327:28) at Object.onReceiveStatus (/Users/milan/Projects/firebase/motus/functions/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)

No workaround found yet that works for rme.

milan-tribe-ai on Sep 3, 2020

I don’t know if it helps anyone, because I’m not using grpc-js, but I got an error in my Firebase function logs:

Exception from a finished function: Error: 14 UNAVAILABLE: No connection established

It looks like the function had spontaneously stopped working around 11:30pm last night.

I found this issue thread and ran npm update to update the dependencies. I redeployed, and it started working again.

These were the packages that got updated:

firebase-functions from “^3.6.1” to “^3.7.0”
typescript from “^3.8.0” to “^3.9.5”
also one of the @types packages, but I don’t think that would have affected it.

j127 on Jun 25, 2020

Still having this issue. Clearing the node_modules and reinstalling the deps didn’t work. Any idea on what’s causing this?

rndlaine on Jun 23, 2020

@jhk115 only Firestore

adamkpurdy on Oct 18, 2019

@jhk115 I’m not doing anything with storage. Only Firestore.

LureOfAdventure on Oct 18, 2019

Nope, you were right to do your due diligence. I hadn’t gotten a fully clean state. Looks like the workaround does work for me.

OneHoopyFrood on Oct 21, 2019

I still experience this, running firebase-functions 3.3.0 and firebase-admin 8.6.0. I tried deleting node_modules and package-lock.json and reinstalling, but still got grpc-js 0.6.4…

ewindso on Oct 21, 2019

Updating from "firebase-functions": "^3.2.0" to "firebase-functions": "^3.3.0" and re deploy worked for me ! Thanks everyone.

floriangbh on Oct 19, 2019

It looks promising on our end! Thanks, everyone.

jhk115 on Oct 18, 2019

@adamkpurdy Upgraded also, working for me so far!

michaeltroya on Oct 18, 2019

@jhk115 This is because of a dependency that firestore is using called grpc-js as mentioned by @clytras

@mrmodelo this occurs to every kind of function that involves firestore and it’s related to firestore changing grpc to grpc-js. There is a workaround for other kind of functions as you can see on my initial post, but I can’t find a way to set old grpc for firestrore funtion triggers.

There have been workarounds referenced if you look further back in the thread and to the linked issues on the original comment, however there are still situations where you can not manage this dependency (to our knowledge so far)

mrmodelo on Oct 18, 2019

@mrmodelo this occurs to every kind of function that involves firestore and it’s related to firestore changing grpc to grpc-js. There is a workaround for other kind of functions as you can see on my initial post, but I can’t find a way to set old grpc for firestrore funtion triggers.

clytras on Oct 18, 2019