netty: Memory leak in latest netty version.

After recent update to 4.1.7-Final (from 4.1.4-Final) my servers started dying with OOM within few hours. Before they were running for weeks with no issues.

Error :

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 64 byte(s) of direct memory (used: 468189141, max: 468189184)
        at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:614) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:568) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:625) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:65) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:131) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:73) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.channel.RecvByteBufAllocator$DelegatingHandle.allocate(RecvByteBufAllocator.java:124) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:956) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:359) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:306) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877) ~[server-0.22.0-SNAPSHOT.jar:?]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) ~[server-0.22.0-SNAPSHOT.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

Or :

08:28:00.752 WARN  - Failed to mark a promise as failure because it has succeeded already: DefaultChannelPromise@7cd20
32d(success)io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 18713 byte(s) of direct memory (used: 468184872, max: 468189184)        
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:631) ~[server-0.21.7-2.jar:?]        
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:585) ~[server-0.21.7-2.jar:?]        
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[server-0.21.7-2.jar:?]        
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:624) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:65) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1533) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1544) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:575) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:550) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:531) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:1324) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1307) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.close(SslHandler.java:498) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:504) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:315) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466) ~[server-0.21.7-2.jar:?]
        at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleUnexpectedException(DefaultExceptionHandler.java:59) ~[server-0.21.7-2.jar:?]
        at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleGeneralException(DefaultExceptionHandler.java:43) ~[server-0.21.7-2.jar:?]
        at cc.blynk.core.http.handlers.StaticFileHandler.exceptionCaught(StaticFileHandler.java:277) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedC:
08:28:00.752 WARN  - Failed to mark a promise as failure because it has succeeded already: DefaultChannelPromise@7cd20
32d(success)io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 18713 byte(s) of direct memory (used: 468184872, max: 468189184)        
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:631) ~[server-0.21.7-2.jar:?]        
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:585) ~[server-0.21.7-2.jar:?]        
	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30) ~[server-0.21.7-2.jar:?]        
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:624) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:65) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[server-0.21.7-2.jar:?]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:1533) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:1544) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:575) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:550) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:531) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.flush(SslHandler.java:1324) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1307) ~[server-0.21.7-2.jar:?]
        at io.netty.handler.ssl.SslHandler.close(SslHandler.java:498) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.close(CombinedChannelDuplexHandler.java:504) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.CombinedChannelDuplexHandler.close(CombinedChannelDuplexHandler.java:315) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelDuplexHandler.close(ChannelDuplexHandler.java:73) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:625) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:609) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466) ~[server-0.21.7-2.jar:?]
        at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleUnexpectedException(DefaultExceptionHandler.java:59) ~[server-0.21.7-2.jar:?]
        at cc.blynk.server.core.protocol.handlers.DefaultExceptionHandler.handleGeneralException(DefaultExceptionHandler.java:43) ~[server-0.21.7-2.jar:?]
        at cc.blynk.core.http.handlers.StaticFileHandler.exceptionCaught(StaticFileHandler.java:277) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:286) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:265) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:257) ~[server-0.21.7-2.jar:?]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedC:

I did restart and made heap dump before abnormal memory consumption and after first error messages from above :

memory

This screenshot shows difference between heap after server start (takes 17% of RAM of Instance) and first OOM in logs (takes 31% of RAM of instance). Instance RAM is 2 GB. So look like all direct memory was consumed (468MB) while heap itself takes less than direct buffers. Load on server is pretty low - 900 req/sec, with ~600 active connections. CPU consumption is only ~15%.

I tried to analyze heap dump but I don’t know netty well in order to make any conclusions.

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

        <netty.version>4.1.7.Final</netty.version>
        <netty.tcnative.version>1.1.33.Fork25</netty.tcnative.version>

        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-transport-native-epoll</artifactId>
            <version>${netty.version}</version>
            <classifier>${epoll.os}</classifier>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative</artifactId>
            <version>${netty.tcnative.version}</version>
            <classifier>${epoll.os}</classifier>
        </dependency>

Right now I’m playing with

-Dio.netty.leakDetectionLevel=advanced 
-Dio.netty.noPreferDirect=true 
-Dio.netty.allocator.type=unpooled 
-Dio.netty.maxDirectMemory=0

to find out working settings. I’ll update ticket with additional info if any.

Unfortunately I wasn’t able to reproduce this issue on QA env. Please let me know if you need more info.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 123 (122 by maintainers)

Most upvoted comments

With the help of @doom369 I was able to track down the change that is guilty for this “regression”. https://github.com/netty/netty/pull/6252 should fix this. The commit message of https://github.com/netty/netty/pull/6252 should give you a better idea what happened, so I will not repeat it here but to make it short its not a memory leak but just a change in how much memory is used when using our custom SSLEngine impl.

@rkapsi @doom369 can you please check the PR and let me know…

Thanks again to @doom369 for all the help tracking this down. Without you this would have been not possible or took a way longer.

normanmaurer on Jan 19, 2017

@doom369 awesome let me try 😃

normanmaurer on Jan 17, 2017

@normanmaurer @Scottmitch So here is my findings so far :

I made a simple test that creates 400 users (opens 400 SSL keep-alive connections) and 400 hardwares (opens 400 plain tcp/ip keep-alive connections). 800 in total. All hardware connections send 1 message in loop. Pipeline on server delivers those message to corresponding user. So this is like 1 to 1 chat for 400 users.

I run same test on few configurations :

epoll + opensll - failed with OOM (1-2 minutes). epoll + no openssl - failed with OOM. epoll + opensll + -Dio.netty.noPreferDirect=true -Dio.netty.allocator.type=unpooled works for ~5 minutes while consumes memory slowly. After it grabs 40% of available memory all connections dropped. However no OOM here. epoll + -Dio.netty.noPreferDirect=true -Dio.netty.allocator.type=unpooled - ok.

Test creates very high request rate (however bandwidth is low ~10 Mbps) so in all tests some of 800 connections are dropped. In average there only 400-500 connections only survived. With low request rate I’m not able to reproduce OOM. On prod load is 10 times lower, so issue for sure not in request rate.

Test code :

    @Test
    public void testConnectAppAndHardware() throws Exception {
        int clientNumber = 400;
        ExecutorService executorService = Executors.newFixedThreadPool(4);

        ClientPair[] clients = new ClientPair[clientNumber];
        List<Future<ClientPair>> futures = new ArrayList<>();

        long start = System.currentTimeMillis();
        for (int i = 0; i < clientNumber; i++) {
            String usernameAndPass = "dima" + i +  "@mail.ua 1";

            Future<ClientPair> future = executorService.submit(
                    () -> initClientsWithSharedNio("xx.xx.xx.xx", 8443, 8442, usernameAndPass, null, properties)
            );
            futures.add(future);
        }

        int counter = 0;
        for (Future<ClientPair> clientPairFuture : futures) {
            clients[counter] = clientPairFuture.get();
            //removing mocks, replace with real class
            clients[counter].appClient.replace(new SimpleClientHandler());
            clients[counter].hardwareClient.replace(new SimpleClientHandler());
            counter++;
        }

        System.out.println(clientNumber + " client pairs created in " + (System.currentTimeMillis() - start));
        assertEquals(clientNumber, counter);

        int i = 0;
        while (true) {
            if (++i % 2 == 0) {
                continue;
            }
            for (ClientPair clientPair : clients) {
                //clientPair.appClient.send("hardware aw 10 10");
                clientPair.hardwareClient.send("hardware vw " + ThreadLocalRandom.current().nextInt(1, 128) + " " + ThreadLocalRandom.current().nextFloat());
            }
            sleep(20);
        }
    }

I tried also to use

-Dio.netty.noUnsafe=true and -Dio.netty.tryUnsafe=false

but in both cases I get NPE :

root@load-test:~# Exception in thread "main" java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:58)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:100)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:87)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:64)
	at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:48)
	at cc.blynk.server.TransportTypeHolder.<init>(TransportTypeHolder.java:43)
	at cc.blynk.server.TransportTypeHolder.<init>(TransportTypeHolder.java:32)
	at cc.blynk.server.Holder.<init>(Holder.java:99)
	at cc.blynk.server.launcher.ServerLauncher.start(ServerLauncher.java:71)
	at cc.blynk.server.launcher.ServerLauncher.main(ServerLauncher.java:65)
Caused by: java.lang.NullPointerException
	at io.netty.util.internal.PlatformDependent0.allocateMemory(PlatformDependent0.java:606)
	at io.netty.util.internal.PlatformDependent.allocateMemory(PlatformDependent.java:264)
	at io.netty.channel.epoll.IovArray.<init>(IovArray.java:64)
	at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:54)
	at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:130)
	at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	... 12 more

Advanced mode for memory leak shows nothing in all cases.

doom369 on Jan 17, 2017

@normanmaurer this is https://github.com/netty/netty/commit/fc3c9c9523150190760801dd0fbf014909519942

Now for sure 😃.

doom369 on Jan 18, 2017

@doom369 thanks for the info with boringssl… I may have an idea. Stay tuned.

normanmaurer on Jan 18, 2017

@doom369 fyi the reason for the double java system property is because they are named differently between 4.0 and 4.1 branch.

johnou on Jan 17, 2017