google-cloud-go: pubsub: Seeing many failed acks despite large MaxExtension and Synchronous

Client

PubSub

Describe Your Environment

fa2eb35 on GCE CentOS 7

Expected Behavior

See very minimal failed acks because expired.

Actual Behavior

Seeing lots (as in greater than 50%) of failed acks because expired.

Settings:

sub.ReceiveSettings.MaxOutstandingMessages = 3
sub.ReceiveSettings.MaxExtension = time.Hour
sub.ReceiveSettings.NumGoroutines = 1
sub.ReceiveSettings.Synchronous = true

The Ack Deadline for this subscription is 10 seconds.

Number of ack’d jobs per minutes:

Number of failed ack’s because expired (via stackdriver) per minute:

We’re ack’ing ~100/minute and stackdriver is reporting that 50-70 of those are ultimately being ignored because the deadline expired.

Time-to-ack 95th percentile:

For some reason there’s only ~20 ModAcks being sent per minute:

But since the 50th percentile is <20 seconds maybe it’s finishing quick enough not to warrant a modack?

We’re acking most messages in less than a minute and since we have MaxExtension set to an hour I’d assume we’d have relatively 0 failed acks.

I’ll be testing to see what happens if I disable Synchronous and if I increase the Ack Deadline to 60 seconds on the subscription.