google-cloud-python: Pubsub 0.35.3 stop consuming messages - ConsumeBidirectionalStream exited RPC

After an ~hour, the rpc connection is closed:

[2018-06-06 11:02:36,258 - INFO - Thread-14 - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,259 - INFO - Thread-ConsumeBidirectionalStream - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,260 - INFO - Thread-14 - streaming_pull_manager] Observed recoverable stream error 504 Deadline Exceeded
[2018-06-06 11:02:36,260 - **ERROR - Thread-ConsumeBidirectionalStream - bidi] The bidirectional RPC unexpectedly exited. <google.api_core.grpc_helpers._StreamingResponseIterator object at 0x7fb758946090>**

I need to cancel and start the subscriber again.

OS type and version - “Debian GNU/Linux 8 (jessie)” Version - Python 2.7.9 4. pip freeze | grep pubsub google-cloud-pubsub==0.35.3

Code example:

        def subscriber_handler(message):
            path = message.attributes.get("type")
            self.handler_function(path=path, body=message.data)  
            message.ack()
        self.listening_subscriber = self.subscriber.subscribe(subscription_path, callback=subscriber_handler, flow_control=flow_control)

All the other info is the same as in https://github.com/GoogleCloudPlatform/google-cloud-python/issues/5325.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 16 (11 by maintainers)

Most upvoted comments

I upgraded to 35.4 and i am still having the issue where my subscriber stops pulling from a subscription. My environment is in kubernetes and should be pulling 40-50 messages a second. Looking at the logs i get the following:

Jun 26, 2018, 12:58:14 PM | INFO:google.cloud.pubsub_v1.subscriber._protocol.leaser:Thread-LeaseMaintainer exiting. INFO:google.cloud.pubsub_v1.subscriber._protocol.heartbeater:Thread-Heartbeater exiting. INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded

After this it seems to go into an endless loop of:

INFO:google.cloud.pubsub_v1.subscriber._protocol.bidi:Re-established stream INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded

Every ten minutes after that it re-establishes the stream then immediately gets the 504 error without pulling any more messages from the subscription. Any ideas?

seanmt13 on Jun 26, 2018

0.35.4 has been released which resolves this as far as I’m able to test. Please let me know if you’re able to reproduce this with that version and we’ll continue digging.

theacodes on Jun 6, 2018