grpc-node: Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: read ECONNRESET

i reuse a connection for all request , sometime seems connection will disconnect when idle for a long times (client error: read ECONNRESET), but the second time visit will be ok

how to set grpc-js for auto reconnect to avoid this error ?

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: read ECONNRESET
    at Object.callErrorFromStatus (/api/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/api/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/api/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/api/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /api/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (node:internal/process/task_queues:76:11)
 {
  code: 13,
  details: 'Received RST_STREAM with code 2 triggered by internal client error: read ECONNRESET',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 14
Comments: 23 (9 by maintainers)

Most upvoted comments

I had exact same issue.

D 2022-05-25T09:19:37.476Z | call_stream | [14] ended with status: code=13 details="Received RST_STREAM with code 2 triggered by internal client error: Protocol error"

I didn’t think the contents of the .env-file was deployed while I was deploying, but turns out firebase takes the contents of it:

FIRESTORE_EMULATOR_HOST="localhost:8080"
PUBSUB_EMULATOR_HOST="localhost:8085"
PUBSUB_PROJECT_ID="myrandomproject-local"

😱 These env-vars certainly shouldn’t have been deployed and the error makes sense. That’s why the Functions timed out when trying to connect with Firestore in my case. It is interesting that it times out rather than promptly failing.

+10

hfossli-agens on May 27, 2022

We have been facing the same issue quite often while using firestore, and the severity is very high as it ends up with data loss or unacceptable latency times most of the time. You can find detailed logs here if it’d help: https://github.com/googleapis/nodejs-firestore/issues/1023#issuecomment-894821062 @ googleapis/nodejs-firestore#1023

pROFESOR11 on Aug 8, 2021

@pROFESOR11 This problem is likely a bad interaction between how gRPC handles connections and how Firebase Cloud Functions networking works. I don’t expect to have a fix in the near future, but I can recommend the following to mitigate the problem:

Set a deadline based on the maximum latency that you can tolerate. It’s probably better to have the client get a “request timed out” error after 30 seconds or whatever and then retry than to wait 3 minutes for the response. Alternatively, you can set the deadline to about half the maximum latency you can tolerate and retry immediately in the function.
Enable keepalives by setting the grpc.keepalive_time_ms and grpc.keepalive_timeout_ms options to enable keepalive functionality. This will help the client detect that the server is unavailable more quickly. Importantly, on a serverless system like Firebase Cloud Functions, you should not set the option grpc.keepalive_permit_without_calls.

It seems that people commonly set grpc.keepalive_time_ms to 10000 (10 seconds) and grpc.keepalive_timeout_ms to 5000 (5 seconds), but the best values may change depending on the server you are talking to. For this mitigation to work optimally, the timeout you use to set the deadline should be at least slightly greater than the sum of those two values. One thing to watch for is if you start getting log lines that say “Connection to [address] rejected by server because of excess pings”, that means that your setting for grpc.keepalive_time_ms is too low.

murgatroid99 on Aug 19, 2021

@murgatroid99 Thanks for your comment. As I mentioned in my previous comment, we’re getting this error on firestore & firebase cloud functions. Do you think if the root cause is something firebase should fix, or it is an grpc-js related issue might be fixed in one of the next versions, or something we should fix in our client / cloud functions code? This issue became a blocker for us, so any help will be much appreciated, thanks.

pROFESOR11 on Aug 18, 2021