aws-sdk-java-v2: closeEventLoopUninterruptibly Hanging on WebFlux Netty

When killing my Spring WebFlux on Netty app that uses S3AsyncClient the app hangs for 16 seconds.

Expected Behavior

Expect the server to be killed faster than 16 seconds.

Current Behavior

After timing out the following stack trace is given:

02:05:04.281 [Thread-19] ERROR s.a.a.h.n.n.NettyNioAsyncHttpClient - Unable to shutdown event loop
java.lang.RuntimeException: Shutting down Netty EventLoopGroup did not complete within 16 seconds
	at software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.closeEventLoopUninterruptibly(NettyNioAsyncHttpClient.java:235)
	at software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.lambda$close$2(NettyNioAsyncHttpClient.java:222)
	at software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient$$Lambda$1600/736839034.run(Unknown Source)
	at software.amazon.awssdk.utils.FunctionalUtils.runAndLogError(FunctionalUtils.java:40)
	at software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.close(NettyNioAsyncHttpClient.java:221)
	at software.amazon.awssdk.utils.IoUtils.closeQuietly(IoUtils.java:73)
	at software.amazon.awssdk.utils.IoUtils.closeIfCloseable(IoUtils.java:90)
	at software.amazon.awssdk.utils.AttributeMap.lambda$close$0(AttributeMap.java:86)
	at software.amazon.awssdk.utils.AttributeMap$$Lambda$1596/1964394254.accept(Unknown Source)
	at java.util.HashMap$Values.forEach(HashMap.java:972)
	at software.amazon.awssdk.utils.AttributeMap.close(AttributeMap.java:86)
	at software.amazon.awssdk.core.client.config.SdkClientConfiguration.close(SdkClientConfiguration.java:79)
	at software.amazon.awssdk.core.internal.http.HttpClientDependencies.close(HttpClientDependencies.java:88)
	at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient.close(AmazonAsyncHttpClient.java:78)
	at software.amazon.awssdk.core.client.handler.BaseAsyncClientHandler.close(BaseAsyncClientHandler.java:138)
	at software.amazon.awssdk.services.s3.DefaultS3AsyncClient.close(DefaultS3AsyncClient.java:3942)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:337)
	at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:271)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:954)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:961)
	at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1041)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1017)
	at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:937)

Steps to Reproduce (for bugs)

  • Deploy any Spring WebFlux 2.0.4.RELEASE app locally
  • Make a request that uses the S3AsyncClient to populate the response
  • Kill the server via control + c

Your Environment

  • AWS Java SDK version used: 2.5.1
  • JDK version used: 1.8.0_51
  • Operating System and version: MacOS High Sierra 10.13.6
  • Spring version: 2.0.4.RELEASE

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 17 (11 by maintainers)

Most upvoted comments

Open a new issue and provide repo code if possible.

Did more research on this and the root cause is actually a race condition in Netty where the event loop gets terminated before channel is closed so globalEventExecutor will be stuck forever waiting for eventloop to notify the channel.close future. This is an known issue with Netty. https://github.com/netty/netty/issues/8398

Thank you for the information! I was not able to reproduce the issue by making the same call and then closing the client.

I wonder if it’s something related to Spring WebFlux. I will try to test with Spring WebFlux.