netty: OOM due to SSL key materials cached every time when there is new connection when using OpenSslCachingX509KeyManagerFactory

Expected behavior

Recently, we found the OOM issue when switching from JDK Ssl to OpenSsl in netty.

We’re using OpenSslCachingX509KeyManagerFactory explicity, so Netty will use OpenSslCachingKeyMaterialProvide to cache and reduce the overhead of parsing the chain and the key for generation of the material.

We expect to see performance optimization but shouldn’t see OOM issue.

Actual behavior

But with stress test for TLS connection, we saw the memory linearly increasing and eventually OOM.

After debugging into the Netty and OpenJDK ssl code, we found the problem is that every time when there is a new connection, handshake cert selection callback OpenSslClientCertificateCallback is called and it will try to find the alias key materials from the cache, if it doesn’t exist it will try to find the match alias from server cert chain, which created a new alias in format of seq_id.builderIndex.keyStoreAlias, like 924450.0.key. And it will parse the chain and key, put into the cache with the new alias, and this retained the refCnt of the key material and prevented the native memory being destroyed, that’s why we eventually saw the OOM issue.

Changing to use OpenSslX509KeyManagerFactory solved this problem.

Steps to reproduce

Using OpenSslCachingX509KeyManagerFactory to set up the SSLContext, and keep issuing Issuing lots of TLS connection requests.

Minimal yet complete reproducer code (or URL to code)

Netty version

We’re using 4.1.36.Final.

JVM version (e.g. java -version)

java version “10.0.1” 2018-04-17 Java™ SE Runtime Environment 18.3 (build 10.0.1+10) Java HotSpot™ 64-Bit Server VM 18.3 (build 10.0.1+10, mixed mode)