spring-cloud-gateway: StacklessClosedChannelException on successfull kubernetes probes

Describe the bug We are using Spring Cloud 2021.0.4 and since a long time we see io.netty.channel.StacklessClosedChannelException at almost every kubernetes probe. The probes return HTTP-Status 200 so we are interested in why this Exceptions are thrown despite the probe has been successfull. They don’t have a message so we get no more information than the stacktrace. We can’t figure out the problem based on the stacktrace.

I understood that a StacklessClosedChannelException can occur when a write to a closed channel/connection happens. As the kubernetes probes succeed we think this should not be the case. As mentioned in this comment https://github.com/netty/netty/issues/12170#issuecomment-1105112857 Netty cannot do anything to this and it could a issue in the code calling netty. As we use the default readiness and liveness probe it can only be an issue in Spring Cloud Gateway or the configuration.

Is there a solution to this or should we just ignore this Exception?

The stacktrace we get:

io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:619)
io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1352)
io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622)
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606)
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:505)
io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:77)
io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:316)
io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:622)
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:606)
io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472)
io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957)
io.netty.channel.AbstractChannel.close(AbstractChannel.java:244)
io.netty.channel.ChannelFutureListener$1.operationComplete(ChannelFutureListener.java:44)
io.netty.channel.ChannelFutureListener$1.operationComplete(ChannelFutureListener.java:41)
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
io.netty.channel.epoll.AbstractEpollChannel.doWriteBytes(AbstractEpollChannel.java:364)
io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytes(AbstractEpollStreamChannel.java:260)
io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteSingle(AbstractEpollStreamChannel.java:471)
io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:429)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:931)
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:895)
io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:531)
io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)
io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:356)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)

Sample We use basic readiness and liveness probes. We see the Exception in Dynatrace.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 2
  • Comments: 15 (3 by maintainers)

Most upvoted comments

Same issue with spring-webflux - reactor-core (3.4.29) - readiness/liveness probes as well, and sometimes with functional service calls as well.

+1 We are seeing the same exception in Dynatrace.