spring-framework: SimpMessagingTemplate.convertAndSend causes hanging thread

Affects: 5.3.3.RELEASE

I’m using spring boot 2.4.2 with RabbitMQ as message broker and spring-boot-starter-jetty. I got an issue on my production server with a jetty thread hangs so long time(almost 2 days) with the thread dump information as below

"qtp682812632-42" Id=0x2a WAITING on java.util.concurrent.CompletableFuture$Signaller@1601acdf
          at java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
          -  waiting on java.util.concurrent.CompletableFuture$Signaller@1601acdf
          at java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
          at java.base@11.0.9.1/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1796)
          at java.base@11.0.9.1/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
          at java.base@11.0.9.1/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1823)
          at java.base@11.0.9.1/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1998)
          at org.springframework.util.concurrent.CompletableToListenableFutureAdapter.get(CompletableToListenableFutureAdapter.java:99)
          at org.springframework.messaging.simp.stomp.StompBrokerRelayMessageHandler$SystemSessionConnectionHandler.forward(StompBrokerRelayMessageHandler.java:1081)
          at org.springframework.messaging.simp.stomp.StompBrokerRelayMessageHandler.handleMessageInternal(StompBrokerRelayMessageHandler.java:599)
          at org.springframework.messaging.simp.broker.AbstractBrokerMessageHandler.handleMessage(AbstractBrokerMessageHandler.java:262)
          at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:144)
          at org.springframework.messaging.support.ExecutorSubscribableChannel.sendInternal(ExecutorSubscribableChannel.java:100)
          at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:139)
          at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:125)
          at org.springframework.messaging.simp.SimpMessagingTemplate.sendInternal(SimpMessagingTemplate.java:187)
          at org.springframework.messaging.simp.SimpMessagingTemplate.doSend(SimpMessagingTemplate.java:162)
          at org.springframework.messaging.simp.SimpMessagingTemplate.doSend(SimpMessagingTemplate.java:48)
          at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
          at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:151)
          at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:129)
          at org.springframework.messaging.core.AbstractMessageSendingTemplate.convertAndSend(AbstractMessageSendingTemplate.java:122)

For sending messages, I just using this code:

response.getMessageList().forEach(m -> messageTemplate.convertAndSend(m.getTopicUrl(), m.getMessageData()));
response.setMessageList(null); //Remove message list
return response;

Any help would be appreciated.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 48 (12 by maintainers)

Most upvoted comments

On a separate note - should the future.get() really be unbound as it is today? Seems MessageDeliveryException() is a viable output of the forward(), so… in my mind waiting for the ChannelFuture to resolve should be bound and if a (configurable) timeout is exceeded, a MessageDeliveryException should be raised…

While this still doesn’t answer why netty doesn’t complete the futures sometimes, it would at least deterministically prevent app freezing, which is still a good thing

Thoughts on that @violetagg @rstoyanchev ?

just a quick update - couldn’t find any way to get the wirelogs out of spring-messaging, so… I built it from source and altered the TcpClient instantiation according to the docs (setting wiretap(true)), now I have these, yay

I don’t think they’ll be of any help really, but at least now I can instrument the entire thing and root cause the problem. I’ll get to the bottom of it, one way or another and post here for everyone’s benefit

I have created a simple project which can reproduce this problem easily. I will upload it here tomorrow because now I’m trying to find the root cause.

I have reverted the change 7758ba3 in spring-boot version 2.5.1(spring version 5.3.8) and found that it worked fine although the thread was in sleeping mode occasionally during the test. So I think there is a concurrent problem in CompletableToListenableFutureAdapter class under the load.

public CompletableToListenableFutureAdapter(CompletableFuture<T> completableFuture) {
		this.completableFuture = completableFuture;
		this.completableFuture.whenComplete((result, ex) -> {
			if (ex != null) {
				this.callbacks.failure(ex);
			}
			else {
				this.callbacks.success(result);
			}
		});
	}

In class CompletableToListenableFutureAdapter, whenComplete function will never been run if there is exception in mono.toFuture() of Mono class.

	@Override
	@SuppressWarnings("unchecked")
	public final void subscribe(Subscriber<? super T> actual) {
		CorePublisher publisher = Operators.onLastAssembly(this);
		CoreSubscriber subscriber = Operators.toCoreSubscriber(actual);

		try {
			if (publisher instanceof OptimizableOperator) {
				OptimizableOperator operator = (OptimizableOperator) publisher;
				while (true) {
					subscriber = operator.subscribeOrReturn(subscriber);
					if (subscriber == null) {
						// null means "I will subscribe myself", returning...
						return;
					}

					OptimizableOperator newSource = operator.nextOptimizableSource();
					if (newSource == null) {
						publisher = operator.source();
						break;
					}
					operator = newSource;
				}
			}

			publisher.subscribe(subscriber);
		}
		catch (Throwable e) {
			Operators.reportThrowInSubscribe(subscriber, e);
			return;
		}
	}

In onError function of MonoToCompletableFuture class, I think ref.getAndSet(null) is null so completeExceptionally is not fired which cause future.get() never been resolved.

	@Override
	public void onError(Throwable t) {
		if (ref.getAndSet(null) != null) {
			completeExceptionally(t);
		}
	}