firebase-admin-node: Firestore: Deadline Exceeded, retrying doesn't help

[READ] Step 1: Are you in the right place?

For issues or feature requests related to the code in this repository file a Github issue.
- If this is a feature request make sure the issue title starts with “FR:”.
For general technical questions, post a question on StackOverflow with the firebase tag.
For general Firebase discussion, use the firebase-talk google group.
For help troubleshooting your application that does not fall under one of the above categories, reach out to the personalized Firebase support channel.

[REQUIRED] Step 2: Describe your environment

Operating System version: Ubuntu 18.04.1 LTS or macOS Mojave
Firebase SDK version: firebase-admin@^6.2.0
Library version: firebase-admin@^6.2.0
Firebase Product: Firestore

[REQUIRED] Step 3: Describe the problem

We are running a server on AWS (though this reproduces locally on a MacBook Pro, also). It makes a series of (mostly) sequential operations on Firestore, including onSnapshot, get, set, and delete. When we run it with a reduced data set, it works well. However, when a larger data set is used, we get the DEADLINE_EXCEEDED error after a long delay (a minute or more). Retrying when this error occurs doesn’t fix the problem, the next call also takes a long time and throws DEADLINE_EXCEEDED.

Sometimes a new data point comes in and it is processed in parallel (they are mostly batched, so mostly sequential). The new data point operates on a different Firestore document than the data point that keeps failing. Surprisingly, the new data point is processed successfully on the first try, while the data point that keeps failing continues failing over and over.

Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
    at Object.exports.createStatusError (/home/ubuntu/rectecgrills-alarms/node_modules/grpc/src/common.js:87:15)
    at ClientReadableStream._emitStatusIfDone (/home/ubuntu/rectecgrills-alarms/node_modules/grpc/src/client.js:235:26)
    at ClientReadableStream._receiveStatus (/home/ubuntu/rectecgrills-alarms/node_modules/grpc/src/client.js:213:8)
    at Object.onReceiveStatus (/home/ubuntu/rectecgrills-alarms/node_modules/grpc/src/client_interceptors.js:1256:15)
    at InterceptingListener._callNext (/home/ubuntu/rectecgrills-alarms/node_modules/grpc/src/client_interceptors.js:564:42)
    at InterceptingListener.onReceiveStatus (/home/ubuntu/rectecgrills-alarms/node_modules/grpc/src/client_interceptors.js:614:8)
    at /home/ubuntu/rectecgrills-alarms/node_modules/grpc/src/client_interceptors.js:1019:24

Relevant Code:

It usually happens on this call:

document.set({ myDataFieldName: false }, { merge: true })

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 26 (6 by maintainers)

Commits related to this issue

Fixes firebase/firebase-admin-node#499 Previously _initializeStream returned a Promise that indicated that the stream was "released", i.e. that it was was ready for attaching listeners. googleapis/n... — committed to googleapis/nodejs-firestore by wilhuff 5 years ago
Fixes firebase/firebase-admin-node#499 Previously _initializeStream returned a Promise that indicated that the stream was "released", i.e. that it was was ready for attaching listeners. googleapis/n... — committed to googleapis/nodejs-firestore by wilhuff 5 years ago
fix: Fix client pooling for long-lived listens (#614) Fixes firebase/firebase-admin-node#499 Previously _initializeStream returned a Promise that indicated that the stream was "released", i.e. th... — committed to googleapis/nodejs-firestore by wilhuff 5 years ago

Most upvoted comments

The Node 6 deprecation has made the timing of releasing this fix somewhat challenging so my best advice right now is that if this fix is critical to you, you should pin to the commit that implements the fix. If you’re using npm to install you can do something like this:

  "dependencies": {
    "@google-cloud/firestore": "googleapis/nodejs-firestore#479bc9c4847cc2a5632a266da706b349a1b74a41",
    "firebase-admin": "^7.3.0"
  }

Note that order matters, as far as npm is concerned. Also, you may need to remove your node_modules and npm install again before this will work.

yarn will succeed in installing this, but it doesn’t run the prepare script that actually builds the package (see yarnpkg/yarn#5235). In this state it will throw indicating that @google-cloud/firestore/build/src/index.js is missing. Unfortunately, our project is written in typescript, so it’s not directly installable.

wilhuff on May 10, 2019