grpc-node: [grpc-js] Error: 13 INTERNAL: Received RST_STREAM with code 0

Problem description

grpc-js client is throwing the following exception under high load:

ERROR Error: 13 INTERNAL: Received RST_STREAM with code 0
    at Object.callErrorFromStatus (/repro/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:342:141)
    at Object.onReceiveStatus (/repro/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)
    at Http2CallStream.outputStatus (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:117:74)
    at Http2CallStream.maybeOutputStatus (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:156:22)
    at Http2CallStream.endCall (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:142:18)
    at ClientHttp2Stream.<anonymous> (/repro/node_modules/@grpc/grpc-js/build/src/call-stream.js:420:22)
    at ClientHttp2Stream.emit (events.js:314:20)
    at emitCloseNT (internal/streams/destroy.js:81:10) {
  code: 13,
  details: 'Received RST_STREAM with code 0',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Reproduction steps

I have a repro repository in the works and will update with the link.

Environment

OS name, version and architecture: macOS Big Sur v11.0 [20A5343j]
Node version: v14.7.0 (seems to be irrelevant, also reproduces with v10.22.0 and v12.18.3
Node installation method: nvs
If applicable, compiler version: n/a
Package name and version: grpc-js v1.1.3

Additional context

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 26
Comments: 20 (4 by maintainers)

Most upvoted comments

For you guys how are using Kubernetes.

Please check your Istio Sidecar. In my case, the traffic to GRCP client was blocked by the Istio Sidecar.

After I disable the Istio Sidecar with annotation, it’s working fine now.

fauzan-n on Jul 22, 2022

I am seeing this as well. I am running a Typescript React App with a node.js server. The server is throwing this error fairly regularly. Something like once an hour. I also get this similar error: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server but about 20% as frequently.

I had a crash on my site on Monday where I was not able to read or write the database and I am trying to figure out if its related.

These are the 3 types of errors I am receiving of this type: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 2

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 0

eli-stewart on Dec 10, 2020

if it helps, we had these errors when running on GKE (google cloud’s kubernetes engine), when under high load and resulting memory pressure. Essentially, our upper limit on the memory was too low, which caused the workloads to use more memory than its limit, which in turn put memory pressure on the nodes, which in turn made it fall apart…
In the specific case, we had a PubSub listener with surges at times.

chroth7 on Sep 29, 2020

thank you @murgatroid99

I’ll have a repro repo up in a bit. the exception is not being thrown when one switches out the client to use the native deprecated grpc module.

dnalborczyk on Aug 11, 2020

I checked the spec again and I was mistaken. An RST_STREAM is not needed/expected after the server sends trailers.

I notice in your first trace log that the line that corresponds to sending the RST_STREAM comes 6 seconds after the line that corresponds to sending trailers. And it seems to be more related to the BDP probe than to the individual stream. So I would guess that if the stream took longer to complete, the server would send the RST_STREAM before the trailers.

murgatroid99 on Jun 14, 2022

If it will help, I’m developing grpc protocol using nghttp2-lib.(C++)

I found that if server sent ping to client, but client didn’t response ping-ack to server, the server will send RST_STREAM with code 0 after sending the trailer.

weimch on Jun 13, 2022

I faced the same problem.

You need to make sure your call method path starts with ‘/’ just like below.

/<package>.<service>/<method>

Having only <method> in the path might cause this problem.

civilizeddev on Aug 27, 2020