grpc-node: Getting GOAWAY error

From @mkamioner on December 28, 2017 9:33

I have been getting the following error a lot after upgrading to 0.15.0

Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"

It seems that the grpc module is printing it (it is the only place in my code base where this string exists) and pubsub is the only component using grpc… Anyone else having this problem? What is the impact and how can we stop it?

Thanks! Mo

Environment details

OS: Debian 8.10, x86_64 GNU/Linux
Node.js version: 6.12.2
npm version: 3.10.10
@google-cloud/pubsub version: 0.16.1

Steps to reproduce

Unsure, we are subscribing with multiple instances to a very active subscription

Copied from original issue: googleapis/nodejs-pubsub#27

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 7
Comments: 17 (7 by maintainers)

Most upvoted comments

Apologies for reviving but I’m seeing this as well. I’m seeing it in my local development environment running against the cloud pubsub emulator. It looks like it happens at one hour of idle time with the the same topic publishers kept alive in-memory whether or not they’ve been actively used and a single long-lived topic subscription.

Will continue providing updates as I’d really like to see this code live in a GKE production environment in the next couple of weeks. Thanks in advance for your help.

Environment

OS: Docker (FROM node:9)
Node.js: v9.8.0
npm: 5.7.1
google-cloud/pubsub: 0.16.5
grpc: 1.9.1

Trace

E0320 05:05:00.368889271      46 chttp2_transport.cc:1105]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
(node:46) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 channel.error listeners added. Use emitter.setMaxListeners() to increase limit
(node:46) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 channel.ready listeners added. Use emitter.setMaxListeners() to increase limit
events.js:165
      throw er; // Unhandled 'error' event
      ^

Error: Cancelled
    at StreamProxy.onConnectionStatus (/app/node_modules/@google-cloud/pubsub/src/connection-pool.js:267:21)
    at StreamProxy.emit (events.js:180:13)
    at ClientDuplexStream.<anonymous> (/app/node_modules/google-gax/lib/streaming.js:130:17)
    at ClientDuplexStream.emit (events.js:180:13)
    at ClientDuplexStream._emitStatusIfDone (/app/node_modules/grpc/src/client.js:273:10)
    at ClientDuplexStream._readsDone (/app/node_modules/grpc/src/client.js:236:8)
    at readCallback (/app/node_modules/grpc/src/client.js:296:12)
Emitted 'error' event at:
    at ConnectionPool.<anonymous> (/app/node_modules/@google-cloud/pubsub/src/subscription.js:1039:10)
    at ConnectionPool.emit (events.js:180:13)
    at StreamProxy.onConnectionStatus (/app/node_modules/@google-cloud/pubsub/src/connection-pool.js:269:14)
    at StreamProxy.emit (events.js:180:13)
    [... lines matching original stack trace ...]
    at readCallback (/app/node_modules/grpc/src/client.js:296:12)

UPDATE 1

I did an rm -rf node_modules && npm cache clean --force && npm install just to ensure all grpc deps were present … will update if/when problem reoccurs.

UPDATE 2

Since ensuring all necessary deps were present with the npm operations above I’ve encountered no problems with the relevant code executing continuously over the last 24 hours or so. ~It seems like guaranteeing necessary dependencies solves the issue and it may not be a “grpc problem,” though better internal validation and error messages from the grpc lib would help users encountering this problem actually diagnose the real underlying issue of missing deps (rather than assuming it’s a grpc bug).~

I’ll update if I encounter the problem again with all deps installed …

UPDATE 3

I still receive sporadic errors of the following form when I hold open a single pubsub topic subscription for several hours:

E0322 09:47:18.099168287      30 chttp2_transport.cc:1105]   Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"

This feels like an error in the pubsub component because I doubt it should be pinging frequently enough to receive ENHANCE YOUR CALM when there’s only a single active subscription and four or five long-lived topic publishers, but … ¯\_(ツ)_/¯

^ FYI @murgatroid99

rdlowrey on Mar 22, 2018

Any update on this?

mkamioner on Jan 1, 2018