rhea: Reconnect and Open Sender Failure
I’m using rhea to periodically send messages to Azure Service Bus using a long-lived connection (new sender for each message). Service Bus will reset the connection whenever a connection has no links or has idle senders for 60 seconds. This is caught by the reconnect logic, but after about 10-20 minutes of reconnecting every minute, the connection will not create and link senders to the connection (I’m using the default reconnect configuration).
The connection reset look like:
Fri, 28 Sep 2018 20:46:56 GMT rhea:frames [connection-3]:0 <- close#18 {"error":{"condition":"amqp:connection:forced","description":"The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId:026f471d5ede4570bd177e86ceea0a98_G5, SystemTracker:gateway5, Timestamp:9/28/2018 8:46:56 PM"}}
Fri, 28 Sep 2018 20:46:56 GMT rhea:events [connection-3] Connection got event: connection_error
Fri, 28 Sep 2018 20:46:56 GMT rhea:events [connection-3] Connection got event: connection_close
Fri, 28 Sep 2018 20:46:56 GMT rhea:events [b224844d-f9b5-f845-9ace-ac6e1b2ba1e7] Container got event: connection_close
Fri, 28 Sep 2018 20:46:56 GMT rhea:frames [connection-3]:0 -> close#18 {}
Fri, 28 Sep 2018 20:46:56 GMT rhea:reconnect [connection-3] Scheduled reconnect in 100ms
For the first 10 minutes after initially creating the connection, creating a sender looks like:
Fri, 28 Sep 2018 20:04:12 GMT rhea:frames [connection-3]:0 -> attach#12 {"name":"ce4c6dd7-f6ae-b046-ac90-e74033dd31b0","source":[],"target":["orders-qa",2]}
Fri, 28 Sep 2018 20:04:12 GMT rhea:frames [connection-3]:0 <- attach#12 {"name":"ce4c6dd7-f6ae-b046-ac90-e74033dd31b0","role":true,"source":[null,null,null,null,null,null,null,null,null,null,null],"target":["orders-qa",2,null,null,null,null,null],"max_message_size":1048576}
But eventually I try to open a sender on the connection, nothing is debug logged from rhea as a result and I never receive any events associated with opening a sender.
I set an interval timer to log the status of the EndpointState
and the underlying socket but both appear to think the connection is healthy. I also have 2 other connections (connection-1
and connection-2
) which have receivers open for different queues (they don’t suffer from timeouts and remain stable).
Is there some state that I should be monitoring that might explain why opening a sender eventually results in no action by rhea? I really appreciate any advice on what might cause this issue.
About this issue
- Original URL
- State: open
- Created 6 years ago
- Comments: 17 (9 by maintainers)
@mattbetzel - The above logs have the service bus keys in it. Please make sure to remove them from the comment and roll over those keys in the azure portal.