grpc-node: `@grpc/grpc-js` throw 'Received RST_STREAM with code 0' with retry enabled

We’ve been facing this error in our production environment for the past 3 days and it’s occurred roughly 15k times: Error: 13 INTERNAL: Received RST_STREAM with code 1

The error is triggered when executing the following code:

const writeResult = await admin
      .firestore()
      .collection(FirestoreCollections.Users)
      .doc(userID)
      .update(fieldsToUpdate); // This line throws the error

We’ve attempted several methods, but none have resolved the problem:

• Switched the update method to use set and set with {merge: true}, but this didn’t work.
• Created a new cloud function dedicated solely to this method. Surprisingly, it didn’t work either. However, it’s noteworthy that we have several other cloud functions that update user data using the same method, and they function as expected. What’s even more perplexing is that this issue arises in about 60% of our http cloud function calls, and it seems to occur randomly. We haven’t been able to identify a consistent pattern.

Interestingly, everything operates flawlessly in our development project. The only difference is that the development project has a smaller User collection.

@murgatroid99 Is any updates on this issue? We using nodejs v16.20.0 and firebase-admin library

The error this bug is about means that the client did not receive trailers and that it received an RST_STREAM with code 0 (no error).

Any suggestion how 1.9.x could lead this random issues with envoy? It may help to understand in which scenario this is causing the issue.

I presented a theory in a previous comment https://github.com/grpc/grpc-node/issues/2569#issuecomment-1883544814. The problem could also be related to https://github.com/envoyproxy/envoy/issues/30149 as mentioned in https://github.com/grpc/grpc-node/issues/2569#issuecomment-1899554032. But in general, I don’t know why Envoy does what it does; the right people to ask about that are the Envoy maintainers.

@maylorsan Code 1 is different from code 0. Please file a separate issue.

We’ve been facing this error in our production environment for the past 3 days and it’s occurred roughly 15k times: Error: 13 INTERNAL: Received RST_STREAM with code 1

The error is triggered when executing the following code:

const writeResult = await admin
      .firestore()
      .collection(FirestoreCollections.Users)
      .doc(userID)
      .update(fieldsToUpdate); // This line throws the error

We’ve attempted several methods, but none have resolved the problem:

• Switched the update method to use set and set with {merge: true}, but this didn’t work.
• Created a new cloud function dedicated solely to this method. Surprisingly, it didn’t work either. However, it’s noteworthy that we have several other cloud functions that update user data using the same method, and they function as expected. What’s even more perplexing is that this issue arises in about 60% of our http cloud function calls, and it seems to occur randomly. We haven’t been able to identify a consistent pattern.

Interestingly, everything operates flawlessly in our development project. The only difference is that the development project has a smaller User collection.

@murgatroid99 Is any updates on this issue? We using nodejs v16.20.0 and firebase-admin library

Same problem i am facing here. is there any update ?

OK, I see what happened: having retries enabled changed the sequencing of some low-level operations, which caused a different outcome with your specific test setup. With retries disabled, each HTTP/2 stream starts and ends in a single synchronous operation, while with retries enabled, starting and ending the stream are separated into two asynchronous operations. The result is that with retries disabled, each stream starts and ends one at a time, while with retries enabled, all of the streams start, and then all of the streams end. This means that in the “retries disabled” case, the remote end can finish processing the first stream before the last stream even starts, while in the “retries enabled” case, the remote end is guaranteed to see all of the streams open at once. So, the primary difference between the observable outcomes in these two tests is that the “retries enabled” test triggers the “max concurrent streams” limit, and the “retries disabled” test does not. If you instead forced the client to reach the “max concurrent streams” limit by opening streaming requests without closing them, you would see the same error in both cases.

The primary problem here, still, is that when the “max concurrent streams” limit is reached, Istio closes the stream in a way that is not valid within the gRPC protocol. Istio is what needs to be changed here. There may be a configuration option to change how it responds in this situation, but if not, this is a bug in Istio.