conscrypt: Native memory leak introduced between version 1.0.1 and 1.1.0 and remains in the latest release.

Hi, It seems that between version 1.0.1 and 1.1.0 something has changed in conscrypt such that it now appears to leak memory when used as the SSL library with the jetty web server. The latest version of conscrypt also appears to leak memory.

To reproduce it start jetty with conscrypt then send http 1.0 requests, this can be done with curl.

Limit the cipher When only one of the following ciphers exist:

• TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
• TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 the memory leak still appears.

Number of java classes I does not look to ever be garbage collected, I made the JVM allocate a bunch of byte[] held on to by a weak reference and call System.gc a bunch but that did not reduce the RSS of the JVM.

running jmap gives:

 num     #instances         #bytes  class name
----------------------------------------------
 510:           111           5328  org.conscrypt.NativeSslSession$Impl
 692:           111           2664  org.conscrypt.ByteArray
 693:           111           2664  org.conscrypt.NativeRef$SSL_SESSION
1577:             5            400  org.conscrypt.SSLParametersImpl
1751:            13            312  org.conscrypt.HostProperties$Architecture
1805:             3            288  org.conscrypt.ConscryptEngine
1806:            12            288  org.conscrypt.HostProperties$Architecture
...
10127:             1             16  org.conscrypt.PeerInfoProvider$1

It doesn’t look like an insane amount of classes are created.

Compile with debug I have a memory profiler (jemalloc), however it only shows that something in the conscrypt shared library is allocating. I would like to try conscrypt with -g and -O0 so I can hopefully get some line numbers on the allocations. I couldn’t find pre-compiled boringSSL with debug symbols. I attempted to compile it myself, however after tinkering

CMakeLists.txt
-  set(C_CXX_FLAGS "-Werror -Wformat=2 -Wsign-compare -Wmissing-field-initializers -Wwrite-strings")
+  set(C_CXX_FLAGS "-Wformat=2 -Wsign-compare -Wmissing-field-initializers -Wwrite-strings")

I ran:

$ cmake -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE -DCMAKE_ASM_FLAGS=-Wa,--noexecstack -GNinja ..
...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/luke/code/third_party/boringssl/build64

After I ran:

$ ninja
[444/444] Linking CXX executable ssl/ssl_test

I set: BORINGSSL_HOME=/home/luke/code/third_party/boringssl/ I then attempted to build conscrypt, with some minor modifications I replaced all -O3 with -O0 and -g.

./gradlew install
./gradlew build

This resulted in building a linux conscrypt I included that and started jetty web server with that along with jemalloc, however the resulting pdf: jemalloc.pdf doesn’t show anything interesting about conscrypt or boringSSL.

It is not clear what to do next, to help identify why conscrypt is leaking memory.