grpc-node: Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY

Problem description

We currently use @grpc/grpc-js with Envoy in production, and we get this mystifying error Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY occasionally during normal RPCs. We currently only use unary calls and not streams. It might have to do with something when we do a new deployment. Does anyone have any insight on how to debug this error?

Reproduction steps

It’s difficult to reproduce because this doesn’t happen in our staging environment.

Environment

alpine linux 3.6
Node version 10.13
@grpc/grpc-js 0.3.2

Additional context

Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY
    at Http2CallStream.call.on (/home/node/app/node_modules/@grpc/grpc-js/build/src/client.js:88:45)
    at Http2CallStream.emit (events.js:187:15)
    at Http2CallStream.EventEmitter.emit (domain.js:442:20)
    at Http2CallStream.endCall (/home/node/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:66:18)
    at Http2CallStream.<anonymous> (/home/node/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:337:18)
    at Generator.next (<anonymous>)
    at fulfilled (/home/node/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:4:58)

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 22 (9 by maintainers)

Most upvoted comments

Also having the same problem with Firestore onSnapshot listeners.

matej-svejda on Aug 15, 2019

That looks like a bug in @grpc/grpc-js. That error message probably means that we are not properly handling certain kinds of connection-level errors. The error message is confusing because it’s coming from the underlying transport, http2, which doesn’t have the unary/stream distinction.

murgatroid99 on Feb 13, 2019