google-cloud-go: PubSub: streaming pull operation stops for 20mins and resumes without error

I experienced an issue that receiving throughput drops to 0, causing queue increase and resume later in about 30mins. Below are stackdriver metrics screen shot 2017-12-18 at 6 04 11 pm Here you can see streaming operation drops around 5:03pm, and all the way to 0. Then close to 5:24pm it went back to normal (it went really high because we discard stale messages).

Undelivered messages shows no one is consuming the topic screen shot 2017-12-18 at 6 04 48 pm

On publishing side traffic is slowly ramping up but nothing unusual. I shut down the publisher after seeing the issue and that’s why undelivered message remains flat.

The fleet is running on GAE Flex with a fix number of instances. MaxOustandingMessages is set to 100 and the blocking is very unlikely resides in business logic code given that we see no errors and all instance metrics were normal.

I’m trying to get repro but just wanna throw what I have right now here to see if anyone seeing similar issues.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 25 (20 by maintainers)

Commits related to this issue

pubsub: test that Receive handles closed streams correctly Add a test that uses the fake to demonstrate that Receive behaves well even if the server closes the stream. This is a failed attempt to re... — committed to googleapis/google-cloud-go by jba 6 years ago

Most upvoted comments

@jrbury Thank you for the info. This does look like the same bug. This CL should fix this.

pongad on Jan 10, 2018

That CL was submitted on January 12, and we just tagged v0.18.0, which includes it. Are you still experiencing the problem?

jba on Jan 22, 2018