spring-framework: SimpMessagingTemplate.convertAndSend causes hanging thread
Affects: 5.3.3.RELEASE
I’m using spring boot 2.4.2 with RabbitMQ as message broker and spring-boot-starter-jetty. I got an issue on my production server with a jetty thread hangs so long time(almost 2 days) with the thread dump information as below
"qtp682812632-42" Id=0x2a WAITING on java.util.concurrent.CompletableFuture$Signaller@1601acdf
at java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.CompletableFuture$Signaller@1601acdf
at java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.9.1/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
at java.base@11.0.9.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
at java.base@11.0.9.1/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
at java.base@11.0.9.1/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
at org.springframework.util.concurrent.CompletableToListenableFutureAdapter.get(CompletableToListenableFutureAdapter.java:99)
at org.springframework.messaging.simp.stomp.StompBrokerRelayMessageHandler$SystemSessionConnectionHandler.forward(StompBrokerRelayMessageHandler.java:1081)
at org.springframework.messaging.simp.stomp.StompBrokerRelayMessageHandler.handleMessageInternal(StompBrokerRelayMessageHandler.java:599)
at org.springframework.messaging.simp.broker.AbstractBrokerMessageHandler.handleMessage(AbstractBrokerMessageHandler.java:262)
at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:144)
at org.springframework.messaging.support.ExecutorSubscribableChannel.sendInternal(ExecutorSubscribableChannel.java:100)
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:139)
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:125)
at org.springframework.messaging.simp.SimpMessagingTemplate.sendInternal(SimpMessagingTemplate.java:187)
at org.springframework.messaging.simp.SimpMessagingTemplate.doSend(SimpMessagingTemplate.java:162)
at org.springframework.messaging.simp.SimpMessagingTemplate.doSend(SimpMessagingTemplate.java:48)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:151)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:129)
at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:122)
For sending messages, I just using this code:
response.getMessageList().forEach(m -> messageTemplate.convertAndSend(m.getTopicUrl(), m.getMessageData()));
response.setMessageList(null); //Remove message list
return response;
Any help would be appreciated.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 48 (12 by maintainers)
On a separate note - should the future.get() really be unbound as it is today? Seems MessageDeliveryException() is a viable output of the forward(), so… in my mind waiting for the ChannelFuture to resolve should be bound and if a (configurable) timeout is exceeded, a MessageDeliveryException should be raised…
While this still doesn’t answer why netty doesn’t complete the futures sometimes, it would at least deterministically prevent app freezing, which is still a good thing
Thoughts on that @violetagg @rstoyanchev ?
just a quick update - couldn’t find any way to get the wirelogs out of spring-messaging, so… I built it from source and altered the TcpClient instantiation according to the docs (setting wiretap(true)), now I have these, yay
I don’t think they’ll be of any help really, but at least now I can instrument the entire thing and root cause the problem. I’ll get to the bottom of it, one way or another and post here for everyone’s benefit
I have created a simple project which can reproduce this problem easily. I will upload it here tomorrow because now I’m trying to find the root cause.
I have reverted the change 7758ba3 in spring-boot version 2.5.1(spring version 5.3.8) and found that it worked fine although the thread was in sleeping mode occasionally during the test. So I think there is a concurrent problem in CompletableToListenableFutureAdapter class under the load.
In class CompletableToListenableFutureAdapter, whenComplete function will never been run if there is exception in mono.toFuture() of Mono class.
In onError function of MonoToCompletableFuture class, I think ref.getAndSet(null) is null so completeExceptionally is not fired which cause future.get() never been resolved.