netty: OOM when uploading large files

I was recently guarding our app with security checks and decided to limit the size of the http request. After this I found that server dies with OOM when I try to upload few large files.

In my test I:

• Run the server;
• Upload 3 times the same file with 1.4GB size;
• Server throws exceptions below;

java.lang.OutOfMemoryError: Direct buffer memory
	at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:768)
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:744)
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:227)
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:327)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
	at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2120)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1325)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1237)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

Profiles result after 3 uploads:

    34008984   29.13%       65  io.netty.buffer.PooledSlicedByteBuf
    27497824   23.55%       50  io.netty.util.Recycler$DefaultHandle
    17348936   14.86%       36  io.netty.buffer.PooledUnsafeDirectByteBuf
    12710968   10.89%       27  java.nio.DirectByteBuffer
    10978400    9.40%       26  io.netty.handler.codec.http.DefaultHttpContent
     5043456    4.32%       16  javax.net.ssl.SSLEngineResult
     3215928    2.75%        5  byte[]
     1961568    1.68%        4  io.netty.buffer.PoolSubpage[]
     1150160    0.99%        2  java.lang.String
      702952    0.60%        1  java.util.ArrayDeque

Server eats 8 GB after 3 tries.

The main difference I found between OOM server and normal server (server that frees the memory between large file uploads) is HttpObjectAggregator.

Server that has OOM has:

                                .addLast("HttpsObjectAggregator",
                                        new HttpObjectAggregator(holder.limits.webRequestMaxSize, true))
web.request.max.size=5242880

Server that frees the memory between file uploads:

                                .addLast("HttpsObjectAggregator",
                                        new HttpObjectAggregator(Integer.MAX_VALUE, true))

Profiles result after 3 uploads without request size limit:

       bytes  percent  samples  top
  ----------  -------  -------  ---
    19270216   16.74%       95  byte[]
    17301648   15.03%        9  byte[] (out)
    15983536   13.88%       56  io.netty.buffer.CompositeByteBuf$Component
    15472152   13.44%       53  io.netty.handler.codec.http.DefaultHttpContent
    12619656   10.96%       46  long[]
     8130168    7.06%       32  java.lang.Object[]
     7883232    6.85%       29  javax.net.ssl.SSLEngineResult
     3072624    2.67%        8  java.lang.String
     2181960    1.90%        7  int[]
     1889664    1.64%        5  short[]
     1754400    1.52%        8  java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
     1731048    1.50%        8  io.netty.buffer.PooledSlicedByteBuf
     1620920    1.41%        7  java.lang.String[]
     1332704    1.16%        5  io.netty.util.Recycler$DefaultHandle
      978648    0.85%        5  io.netty.util.ResourceLeakDetector$Record

Similar issue - https://github.com/netty/netty/issues/3559.

openjdk version "11.0.1" 2018-10-16
OpenJDK Runtime Environment (build 11.0.1+13-Ubuntu-3ubuntu116.04ppa1)
OpenJDK 64-Bit Server VM (build 11.0.1+13-Ubuntu-3ubuntu116.04ppa1, mixed mode, sharing)

        <netty.version>4.1.34.Final</netty.version>
        <netty.boring.ssl.version>2.0.22.Final</netty.boring.ssl.version>

Pipeline is pretty complex, however this is main handlers:

.addLast(holder.sslContextHolder.sslCtx.newHandler(ch.alloc()))
.addLast("HttpsServerCodec", new HttpServerCodec())
.addLast("HttpsObjectAggregator", new HttpObjectAggregator(holder.limits.webRequestMaxSize, true))
.addLast("HttpsChunkedWrite", new ChunkedWriteHandler())