rhea: Reconnect and Open Sender Failure

I’m using rhea to periodically send messages to Azure Service Bus using a long-lived connection (new sender for each message). Service Bus will reset the connection whenever a connection has no links or has idle senders for 60 seconds. This is caught by the reconnect logic, but after about 10-20 minutes of reconnecting every minute, the connection will not create and link senders to the connection (I’m using the default reconnect configuration).

The connection reset look like:

Fri, 28 Sep 2018 20:46:56 GMT rhea:frames [connection-3]:0 <- close#18 {"error":{"condition":"amqp:connection:forced","description":"The connection was inactive for more than the allowed 60000 milliseconds and is closed by container 'LinkTracker'. TrackingId:026f471d5ede4570bd177e86ceea0a98_G5, SystemTracker:gateway5, Timestamp:9/28/2018 8:46:56 PM"}} 
Fri, 28 Sep 2018 20:46:56 GMT rhea:events [connection-3] Connection got event: connection_error
Fri, 28 Sep 2018 20:46:56 GMT rhea:events [connection-3] Connection got event: connection_close
Fri, 28 Sep 2018 20:46:56 GMT rhea:events [b224844d-f9b5-f845-9ace-ac6e1b2ba1e7] Container got event: connection_close
Fri, 28 Sep 2018 20:46:56 GMT rhea:frames [connection-3]:0 -> close#18 {} 
Fri, 28 Sep 2018 20:46:56 GMT rhea:reconnect [connection-3] Scheduled reconnect in 100ms

For the first 10 minutes after initially creating the connection, creating a sender looks like:

Fri, 28 Sep 2018 20:04:12 GMT rhea:frames [connection-3]:0 -> attach#12 {"name":"ce4c6dd7-f6ae-b046-ac90-e74033dd31b0","source":[],"target":["orders-qa",2]} 
Fri, 28 Sep 2018 20:04:12 GMT rhea:frames [connection-3]:0 <- attach#12 {"name":"ce4c6dd7-f6ae-b046-ac90-e74033dd31b0","role":true,"source":[null,null,null,null,null,null,null,null,null,null,null],"target":["orders-qa",2,null,null,null,null,null],"max_message_size":1048576} 

But eventually I try to open a sender on the connection, nothing is debug logged from rhea as a result and I never receive any events associated with opening a sender.

I set an interval timer to log the status of the EndpointState and the underlying socket but both appear to think the connection is healthy. I also have 2 other connections (connection-1 and connection-2) which have receivers open for different queues (they don’t suffer from timeouts and remain stable).

Is there some state that I should be monitoring that might explain why opening a sender eventually results in no action by rhea? I really appreciate any advice on what might cause this issue.

About this issue

  • Original URL
  • State: open
  • Created 6 years ago
  • Comments: 17 (9 by maintainers)

Most upvoted comments

@mattbetzel - The above logs have the service bus keys in it. Please make sure to remove them from the comment and roll over those keys in the azure portal.