nodejs-pubsub: user witnessing timeouts on @grpc/grpc-js@0.6.9

@ericpearson, I went ahead and opened this issue in reference to the discussion in #790.

I’d like start delving in a bit deeper, so that we can hopefully figure out a pattern in the timeouts you’re seeing.

Environment details

OS/platform: What are you running @google-cloud/pubsub on, Google App Engine, Kubernetes Engine, other infrastructure?
Node.js version: ?
npm version: ?
@google-cloud/pubsub version: ?? (Is PubSub a good place for us to start with debugging, you’d mentioned you’d seen the issue with some other grpc libs, but PubSub tends to be people’s highest workload).

Getting us debug information

If you run one of your clients with the following environment variables set:

export GRPC_TRACE=all
export GRPC_VERBOSITY=DEBUG

or in your application run the following (before your pubsub import):

process.env.GRPC_TRACE = 'all';
process.env.GRPC_VERBOSITY = 'DEBUG';
const {PubSub} = require('@google-cloud/pubsub');

☝️ we will start seeing detailed grpc logs, and should be able to identify exactly what happened immediately before a message timing out.

To get the logs over to me, you can both open an issue:

https://issuetracker.google.com/savedsearches/559741, and send an email my way so that I can make sure the grpc team gets the logs immediately (I’m ben [at] google.com).

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

The “Deadline exceeded” bug is back for me using:

  "dependencies": {
    "@google-cloud/firestore": "^2.6.0",
    "@google-cloud/logging": "5.5.5",
    "@google-cloud/logging-bunyan": "^2.0.0",
    "@google-cloud/pubsub": "1.1.5",

I thought it was fixed? This is the error:

{"ackIds":["KlgRTgQhIT4wPkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUaAiBuFD1CWhoOB1xcdQdRDB8neDB2a1xHUwFPU3RfcysvV1RYcgBWCRtye2d0bFsVAyrBuOnH3ZuSWUZgTc3pmMdHXbXW3-giZh49WxJLLD5-MzVFQV5AEkw4AERJUytDCw"],"code":4,"details":"Deadline exceeded","metadata":{"internalRepr":{},"options":{}}}

Is there any more information I can provide?

npomfret on Nov 11, 2019

@oande I’ve created a tracking issue to add tuning documentation to our README.md:

https://github.com/googleapis/nodejs-pubsub/issues/810

I’m closing this issue for the time being, due to lack of information about the original underlying bug. @ericpearson, please don’t hesitate to re-open if you’re continuing to bump into issue,s and can provide some specific debugging information about the timeouts you’re seeing.

bcoe on Nov 4, 2019

@oande I would probably just increase the timeout myself. It looks like you’re finding yourself with an empty queue and IMO it would be better to wait than to make additional requests that will likely timeout as well.

callmehiphop on Oct 28, 2019