google-cloud-go: pubsub: optionally provide a blocking receiver

@bradleyfalzon The answer is somewhat complex. Please bear with me.

We want to move the client library to streaming eventually, because it will be faster. Unfortunately, in streaming we don’t have complete control over the number of messages we pull. For this reason, we cannot say “only pull 1”.

If you set ReceiveSettings.MaxOutstandingMessages to 5, the client might pull more than 5 messages, but it will take care to never run more than 5 callbacks concurrently. Limiting the number of callbacks makes sure we don’t overwhelm your machines. Since we can pull more messages than we can concurrently process, the latency of some messages will be higher: it’s just waiting around and not delivered to other machines. On the other hand, the overall throughput should improve, since your CPU should spend less time idle. If this trade-off works for you, ReceiveSettings is probably the right choice.

On the other hand, if you require that all pulled messages should be immediate processed and no message should be waiting around, the high-performance client unfortunately doesn’t support this use case. The gist you linked should work well enough though.

Does this answer your question? Please let me know if I can help more!

@bradleyfalzon That’s correct, the client at cloud.google.com/go/pubsub will continue to focus on high-throughput use cases.

By the way, I neglected error-handling in the first version of that gist, so please revisit to get the latest.

@bradleyfalzon We’re still working with the pubsub team on the API issue, but I can comment on your gist specifically.

I think the problem is that runtime.Goexit runs all deferred calls. So it closes the subscriber while the second goroutine is still using it to pull messages. If you just need the main goroutine to hang forever, maybe replace the call with select{}?

What’s the use case?

This decision was discussed extensively internally. We can publish some of the reasoning behind the recent changes.