netty: io.netty.util.internal.OutOfDirectMemoryError

Actual behavior

io.netty.util.internal.OutOfDirectMemoryError - failed to allocate 16777216 byte(s) of direct memory (used: 3456106503, max: 3464495104) 200 B (0%) n/a

Netty version

4.1.16.Final

JVM version (e.g. java -version)

1.8.0_152

OS version (e.g. uname -a)

Linux

Four weeks after netty uptime, I’m experiencing CPU spikes that once happen they stay high. Then, netty server fails to accept an amount of 10-20% of incoming connections. I captured the threads that consume the most cpu and I found the exception they were throwing was io.netty.util.internal.OutOfDirectMemoryError.

Also before this I’m experiencing garbage collection issues once a week, system hangs unless I call manually System.gc() from a remote administration panel that uses another netty bootstrap instance for the admin client. Therefore the memory is heavily used but always goes down after full gc.