nodejs-pubsub: Streaming pull subscriber does not notify of loss of network
- OS: Ubuntu 22.04
- Node.js version: 18.16.0
- npm version: 9.5.1
@google-cloud/pubsubversion: 3.7.1
Steps to reproduce
- Call PubSub.subscription(), and then setup on() handlers for ‘message’, ‘debug’, ‘error’, and ‘close’
- After initiating the initial subscription and receiving some streaming pull messages, disconnect the network on the machine
- Notice that there are no error or debug events of any kind after several minutes. In other words, there is no notice of the fact that the network is gone and the subscription pull is therefore no longer operating. If the network is disconnected before the streaming pull begins, then there are debug messages and eventually there is an error after the default 5 minute timeout where the subscription is closed. But none of this happens if the network disconnects after the stream has been established.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 5
- Comments: 17
Any update on this? We are seeing this quite regularly and it is causing a lot of issues. Mainly because the high volume of messages and a need for manual intervention to re-establish the connection is is a bad combination
Currently using version 4.0.1 in GKE environment
Here is what I’ve found so far: pubsub has a keepalive mechanism it seems internally that writes an empty object to the gRPC stream wrapper every 30 seconds, but this seems as if it will never fail due to the below behavior.
pubsub uses gRPC, which uses Node’s HTTP2 API.
Even that HTTP2 API does not seem to detect loss of network when trying to
write()to theHttp2Stream.Http2Streamuses Node’sstream.Duplexclass internally, which says (via thestream.Writabledocs) that it will buffer writes. It would seem that the call towrite()will buffer and provides no timely or clear indication that the underlying TCP socket is not receiving ack’s and thus disconnected. (there is some more complexity around how it buffers in the docs; I’ve found on Windows that it does behave as documented but not on Linux)Update: the default modern Linux TCP ACK timeout is about ~16 minutes after retries. Only then will the write() indicate an issue unless TCP_USER_TIMEOUT is set on the Socket. See here for some details.
So when gRPC ultimately calls
write(), it also doesn’t know about the disconnection. However, I see that gRPC has its own keepalive mechanism, which seems to use HTTP2 pings under the covers, and those work in this situation. In other words, when HTTP2 pings fail to receive responses, this is easy to detect and react to, which gRPC does. But this is not particularly well exposed to pubsub on top of it.If I pass
{"grpc.keepalive_time_ms": 60000}in thePubSub()constructor options (which is itself seemingly undocumented as a supported parameter), it does activate this system within gRPC and it will send HTTP2 pings and detect the loss of connection within that time as configured.However, it seems the most viable (somewhat hack-ish) way to bubble that up to pubsub is code like this:
And then you can use the connectivity state values to know when it transitions from 2 (“READY”) to 3 (“TRANSIENT FAILIURE”)
watchConnectivityState docs
But I feel like using this undocumented internal API is a hack.
I love the fact that pubsub, gRPC, and actually Node’s HTTP2 APIs all try to seamlessly re-establish connectivity without failing, but when taken to this extreme in a situation where (let’s say) the pubsub Publisher publishes infrequently and unpredictably, it becomes very difficult to detect network issues in a timely way. And if timeliness is important, I don’t know of a solution other than this hack. Would it be not using Streaming Pull? Seems extreme. I have to use pubsub because it’s the only mechanism provided to me (by Google in this case) to get my data (from Google).
So I really think there should be documented public API support for this.
End note: it took me many hours of frustrating debugging to figure this out due to documentations gaps in all of these APIs
Another update: if I override the default “deadline” where streams are restarted every 15 minutes to be some very long time (e.g. hours), then I can get watchConnectivityState() to trigger upon TCP socket failure (at ~16 mins on Linux) even with grpc.keepalive_time_ms not set.
I think what is happening is that if the deadline is shorter than the TCP ACK timeout, the HTTP2 streams are destroyed and re-created prior to the TCP timeout, and so no error is ever generated in those streams, even though they are being written to by the pubsub keepalive.
It is mysterious though that I guess Node ignores these errors from the underlying socket and continues to try to write to it, never propagating them to the Http2Session level, just because the original Http2Stream that was being written to is no longer alive. Because in the end there is just a single socket that is present the whole time.
If the Http2Session was destroyed correctly, then gRPC would have to re-create it, which would certainly fail because at this point no new socket can successfully be opened.