netty: TCN is failing with error:14179044:SSL routines:tls_construct_server_key_exchange:internal error

We’re seeing some ciphers such as TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (a.k.a. DHE-RSA-AES128-GCM-SHA256) failing with TCN 2.0.12.

Here is a very crude repro.

public static void main(String[] args) throws Exception {
    String cipher = "DHE-RSA-AES128-GCM-SHA256";
    
    SelfSignedCertificate cert = new SelfSignedCertificate();
    
    ChannelHandler handler = new ChannelInitializer<Channel>() {
      @Override
      protected void initChannel(Channel ch) throws Exception {
        ChannelPipeline pipeline = ch.pipeline();
        
        SslContext sslContext = SslContextBuilder.forServer(cert.certificate(), cert.privateKey())
          .sslProvider(SslProvider.OPENSSL)
          .ciphers(Arrays.asList(cipher))
          .build();
        
        pipeline.addLast(sslContext.newHandler(ch.alloc()));
        pipeline.addLast(new ChannelInboundHandlerAdapter() {
          @Override
          public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
            if (evt instanceof SslHandshakeCompletionEvent) {
              SslHandshakeCompletionEvent event = (SslHandshakeCompletionEvent)evt;
              if (!event.isSuccess()) {
                Throwable cause = event.cause();
                if (cause instanceof SSLHandshakeException) {
                  cause.printStackTrace();
                }
              }
            }
            
            ctx.fireUserEventTriggered(evt);
          }
          
          @Override
          public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
            cause.printStackTrace();
          }
        });
      }
    };
    
    ServerBootstrap bootstrap = new ServerBootstrap()
        .channel(NioServerSocketChannel.class)
        .group(new NioEventLoopGroup())
        .childHandler(handler);
    
    Channel channel = bootstrap.bind(0).sync().channel();
    InetSocketAddress isa = (InetSocketAddress)channel.localAddress();
    
    System.out.println("Server: " + OpenSsl.versionString());
    System.out.println(cipher + " is supported: " + OpenSsl.isCipherSuiteAvailable(cipher));
    
    System.out.println("Try: openssl s_client -cipher " + cipher + " -connect localhost:" + isa.getPort());
    
    Thread.sleep(Long.MAX_VALUE);
  }

$ openssl s_client -cipher DHE-RSA-AES128-GCM-SHA256 -connect localhost:24619
CONNECTED(00000003)
depth=0 CN = example.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = example.com
verify return:1
140045187453376:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1407:SSL alert number 80
---
Certificate chain
 0 s:/CN=example.com
   i:/CN=example.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBqjCCAROgAwIBAgIJAL5hwsKKvJqWMA0GCSqGSIb3DQEBBQUAMBYxFDASBgNV
BAMTC2V4YW1wbGUuY29tMCAXDTE3MDgwMjAwMzk0NloYDzk5OTkxMjMxMjM1OTU5
WjAWMRQwEgYDVQQDEwtleGFtcGxlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAw
gYkCgYEAk5aJT4NDokMO+YSEuJUmGDxqMEiLV9bVu4SoPP1v/WXPsQfiiscuP3O+
T5lGk9M8PtBD/803jHra4YaFvoW8cNTxk2tBOzq9FadOGEpnuTIfg2RTSpDkH8X+
dMhQfqvUZGSAJ9GmjBZ1mhIF4ECTSoyeuRlagPMcpzgkVuOLN5ECAwEAATANBgkq
hkiG9w0BAQUFAAOBgQB2MmMeqI12VfIQk6bV5ppjYEFay0avSlHeWyExG8Uot9Uv
7j1hynx1dpWcUhH+33HtmHuJSqRy1AhskCLSHNg1LXIJfwZFrZXln2s/1RrA781p
e3mrOSQ47UIVrBRrnk8HV+UFVcZlCJTWY//IKiEQEyoCowzWOLTjxvNMnXxRfA==
-----END CERTIFICATE-----
subject=/CN=example.com
issuer=/CN=example.com
---
No client certificate CA names sent
---
SSL handshake has read 542 bytes and written 102 bytes
Verification error: self signed certificate
---
New, (NONE), Cipher is (NONE)
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000 
    Session-ID: E6E1CEE7455FA36EC45A81F0D1144392ED7A8CDB61F477F26372C60829949171
    Session-ID-ctx: 
    Master-Key: 
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1533170393
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: yes
---

Notice this error in the above output, and Cipher : 0000 and the client exists on its own.

140045187453376:error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:1407:SSL alert number 80

javax.net.ssl.SSLHandshakeException: error:14179044:SSL routines:tls_construct_server_key_exchange:internal error
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1114)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1076)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1143)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1186)
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1275)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1189)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1223)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:14179044:SSL routines:tls_construct_server_key_exchange:internal error
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: error:14179044:SSL routines:tls_construct_server_key_exchange:internal error
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1114)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1076)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1143)
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1186)
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1275)
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1189)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1223)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	... 16 more

The same test when switching to previous TCN version w/ OpenSSL 1.0.2 works as expected.

$ openssl s_client -cipher DHE-RSA-AES128-GCM-SHA256 -connect localhost:27611
CONNECTED(00000003)
depth=0 CN = example.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = example.com
verify return:1
---
Certificate chain
 0 s:/CN=example.com
   i:/CN=example.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBqTCCARKgAwIBAgIIIygVaeXtHowwDQYJKoZIhvcNAQEFBQAwFjEUMBIGA1UE
AxMLZXhhbXBsZS5jb20wIBcNMTcwODAyMDA0MzE3WhgPOTk5OTEyMzEyMzU5NTla
MBYxFDASBgNVBAMTC2V4YW1wbGUuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCB
iQKBgQCYq2Dq6dNAj5HfDv1PYGXY0Of9/mEB1/SV0ap1BRLMkJ11FE4kc+RBYK1X
Fj7z3gT/B9bUcgcBLqd+QtuKiKQaqZ9LyEyMnXuCECp54ATF5cKCo7t17P7UjG5W
W+vG7BcpZgILpIzUOg9uHm9QmQF6N2PhAzyWeOHOygmVfTskHQIDAQABMA0GCSqG
SIb3DQEBBQUAA4GBABsQmq73/GdhijgSF6c/Us0RUKzCxdUNekjsezV8mZ9c0Rkr
cpgHVwA9Kxd2EOTWnWt1OGb0WpUNOmxxRoMvUXLMnIHOvGGfnOYoQUJGBJEpXof/
YK2YqfQD/DufOzVjpHp0/nZfhnNZQr9YiWBdFrP91Ch7vFmAS5MQ/ARNXKaT
-----END CERTIFICATE-----
subject=/CN=example.com
issuer=/CN=example.com
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: DH, 1024 bits
---
SSL handshake has read 994 bytes and written 292 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is DHE-RSA-AES128-GCM-SHA256
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : DHE-RSA-AES128-GCM-SHA256
    Session-ID: A1EE6830C05434ECBE0435046BAE6CFD7F01971B039AF1003072F4EFCCEFE6FA
    Session-ID-ctx: 
    Master-Key: 9309D650B879B99E6BEB7DCCE35819AB17E350D1359F3670205F77E165443FF036F2A4D12107654C1C14C6226FAFDE3B
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1533170605
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
---
^C

Notice that there is no error, Cipher : DHE-RSA-AES128-GCM-SHA256 and I need to exit with ^C as usual.

Expected behavior

Actual behavior

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

Netty version

4.1.28 w/ TCN 2.0.12 (OpenSSL static on Linux)

JVM version (e.g. java -version)

OS version (e.g. uname -a)