reactor-netty: Bytebuf leakage when terminating the stream with a timeout

We are currently running into critical memory issues using reactor netty on our production environment. Our guess is that the issue lies in the stream being terminated before the allocated Bytebufs are released when timeouts are used.

Our application uses reactor-netty to query various JSON web APIs, and those calls are wrapped with a Hystrix Command to manage timeouts and add a circuit breaker. Whenever a HystrixTimeout occurs, the stream on the HTTP client is terminated. However, if reactor-netty is still aggregating the response and thus has ByteBufs allocated, those ByteBufs are (according to the Netty leak detector) not cleaned up.

2017-07-03 14:14:42.234 ERROR 6609 --- [ntLoopGroup-2-7] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.

We’ve created a sample application demonstrating the issue. The sample application uses the standard Reactor timeout instead of Hystrix, but produces the same errors. Running the tests should show the leaks.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 5
Comments: 27 (14 by maintainers)

Most upvoted comments

2018-01-29 19:16:00.996 ERROR 19636 --- [ctor-http-nio-2] r.ipc.netty.channel.ChannelOperations    : [HttpClient] Error processing connection. Requesting close the channel

reactor.core.Exceptions$BubblingException: java.io.IOException: Connection closed prematurely
	at reactor.core.Exceptions.bubble(Exceptions.java:154) ~[reactor-core-3.1.3.RELEASE.jar:3.1.3.RELEASE]
	at reactor.core.publisher.Operators.onErrorDropped(Operators.java:256) ~[reactor-core-3.1.3.RELEASE.jar:3.1.3.RELEASE]
	at reactor.ipc.netty.channel.FluxReceive.onInboundError(FluxReceive.java:348) ~[reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]
	at reactor.ipc.netty.channel.ChannelOperations.onInboundError(ChannelOperations.java:437) ~[reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]
	at reactor.ipc.netty.http.client.HttpClientOperations.onInboundClose(HttpClientOperations.java:265) ~[reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]
	at reactor.ipc.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:110) ~[reactor-netty-0.7.3.RELEASE.jar:0.7.3.RELEASE]

@dancingfrog this means the server closes the connection. If you see other issues please report them as separate issues. If you have questions please use our gitter channel. https://gitter.im/reactor/reactor

I’m closing this issue and restoring the original milestone.

Regards, Violeta

violetagg on Jan 30, 2018

Note that you can do the same without downcasting, i.e.:

} finally {
    DataBufferUtils.release(dataBuffer);
}

rstoyanchev on Jan 29, 2018

My team has decided that if we don’t have a fix on September 1 for this issue, we’ll start migrating our production applications to another non-blocking HTTP client, because we don’t want to keep running with this memory leak and automated restarts in production. We would like to be able to continue using reactor-netty, so is there anything me and my colleagues can do to help fixing this issue? We are not familiar with the reactor-netty codebase, but we are willing to invest some time into getting this fixed, so if you maybe have some pointers for us, we could maybe help?

breun on Aug 21, 2017