nodejs-pubsub: Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

@feywind I just tried both of the suggested workarounds above, we were already using grpc-js@1.8.21 , so I added the grpc config keys are you suggested. Unfortunately, this does not solve the issue. We still see a constant 4 DEADLINE_EXCEEDED error every 20 minutes from our subscriber.

Thanks for the discussion above!

@feywind we have a support contract and have opened a ticket for further investigation, it’s #45302224 if you want to follow along. The commonalities we see with others on this thread is that we create a client for each subscription and listen to about a dozen subscriptions per cloud run instance. Across our 20 instances, we get about 30 errors that repeat on a regular pattern every 15 minutes; these are captured via datadog-agent. So it’s possible this is intentional error throwing and handling based on the message-stream.js code.

If anyone hasn’t tried turning on grpc keepalive support here, please do try that to see if it helps anything. e.g.:

For JavaScript:

const pubsubClient = new PubSub({ 'grpc.keepalive_timeout_ms': 10000, 'grpc.keepalive_time_ms': 30000 });

Or for TypeScript:

const pubsubClient = new PubSub({ 'grpc.keepalive_timeout_ms': 10000, 'grpc.keepalive_time_ms': 30000 } as any);

Might also want to force-upgrade grpc-js in your project:

npm i --save @grpc/grpc-js@1.8.19

These are all just workarounds, but some people are having success. We’re still trying to get a root cause. You’ll probably want to remove that grpc-js package dependency later so you get updates again, and we’re talking about just making keepalives be an always-on thing.

https://github.com/grpc/grpc-node/issues/2690 Might be the cause of your recent DEADLINE_EXCEEDED. Try updating @grpc/grpc-js to 1.10.4

At least regarding unary pull, deadline exceeded errors are going to happen from time to time and probably be even more likely when there are no messages to pull. If there are no messages, it means the request is going to wait the longest to get back a response.

Could you try making DEADLINE_EXCEEDED a retryable error for Pull? Here’s a similar example for specifying custom retries for Publish

@mr-pascal Thanks a lot!

CPU allocation: CPU is always allocated might be not related because we’re already running with that configuration and we’re still getting error.