spring-amqp: rabbitmq listeners are not consuming after connection reset
We have 2 listeners (java processes inside docker on separated machines) for a rabbit queue(on cloudamqp) that were reading from a queue, and then for some reason stopped reading. According to our logs, on September 30th 2018, 11:28:37.848 (UTC) our application experienced a connection reset:
September 30th 2018, 11:28:37.848 impl.ForgivingExceptionHandler (ForgivingExceptionHandler.java:120) - An unexpected connection driver error occurred (Exception message: Connection reset)
September 30th 2018, 11:28:37.849 connection.CachingConnectionFactory$DefaultChannelCloseLogger (CachingConnectionFactory.java:1278) - Channel shutdown: connection error
September 30th 2018, 11:28:37.849 connection.CachingConnectionFactory$DefaultChannelCloseLogger (CachingConnectionFactory.java:1278) - Channel shutdown: connection error
September 30th 2018, 11:28:39.401 connection.AbstractConnectionFactory (AbstractConnectionFactory.java:360) - Created new connection: connectionFactory#623d41c5:2/SimpleConnection@1beb3736 [delegate=amqp://xxxx@54.86.215.255:5672/prod, localPort= 57360]
It seems that since the connection was reset and restored, but we are not getting any new messages from rabbit since then, although there are many messages there. In RabbitMQ Manager I can see the queue size is getting larger but there are no consumers. On thread dump I can see the AMQP thread is running. On netstat I can see connection to cloudamqp is established. We are running a java application with: openjdk version "1.8.0_151" org.springframework.boot:spring-boot-starter-amqp:2.0.4.RELEASE Any help on how to continue investigating. We don’t want to restart the dockers until it is investigated.
Thanks
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 30 (15 by maintainers)
The app must reconnect and re-create the exchange, the queue and the binding between them and everything must be working smoothly when the rabbitmq server is back and working again. But this doesn’t happen. Everything is re-created (connection, exchange, queue and biding) successfully but the queue doesn’t have consumers
I said that the app must re-create everything after the rabbitmq server is back and woking properly again. This means that you need to restore the rabbitmq users after the server is up and running again. Obviously this is done because after the restart of the rabbitmq the java app re-creates the queue, the exchange and binding between them without restarting the java app => the connection is also re-created. The only problem is that the queue doesn’t have consumers after it was re-created