netty: TLS failures (BoringSSL with delegating executor) after updating Netty
I’ve been trying to reproduce this for some time without success… unfortunately we’re not able to reproduce it in staging… thought it was worth sharing all my debugging so far to get some fresh eyes.
Expected behavior
Actual behavior
Here are some samples I pulled from the server logs, quick note is that it is a public facing server… so there are also a lot of bots connecting and probing, but it was evident from client crash logs and metrics that there were problems.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:100003fc:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_RECORD_MAC
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$0(NonBlockingIOThreadFactory.java:18) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$1(NonBlockingIOThreadFactory.java:26) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.70.Final.jar:4.1.70.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: javax.net.ssl.SSLException: error:100003fc:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_RECORD_MAC
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1071) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1365) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1305) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1392) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1435) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
... 24 more
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$0(NonBlockingIOThreadFactory.java:18) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.70.Final.jar:4.1.70.Final]
at com.sulake.h4k.common.util.io.NonBlockingIOThreadFactory.lambda$newThread$1(NonBlockingIOThreadFactory.java:26) ~[h4k-common-7.0.532.jar:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.70.Final.jar:4.1.70.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: javax.net.ssl.SSLException: error:1000042e:SSL routines:OPENSSL_internal:TLSV1_ALERT_PROTOCOL_VERSION
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1071) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1365) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1305) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1392) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1435) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284) ~[netty-handler-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.70.Final.jar:4.1.70.Final]
... 24 more
client (Unity with C#) crash logs
TlsException: Failed to read data to TLS context - error code: UNITYTLS_INTERNAL_ERROR
Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) (at <00000000000000000000000000000000>:0)
Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) (at <00000000000000000000000000000000>:0)
Mono.Unity.UnityTlsContext.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream.ProcessRead (Mono.Net.Security.BufferOffsetSize userBuffer) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncReadRequest.Run (Mono.Net.Security.AsyncOperationStatus status) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<InnerRead>d__25.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<InnerRead>d__66.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) (at <00000000000000000000000000000000>:0)
System.AsyncCallback.Invoke (System.IAsyncResult ar) (at <00000000000000000000000000000000>:0)
System.Threading.WaitCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ThreadPoolWorkQueue.Dispatch () (at <00000000000000000000000000000000>:0)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<StartOperation>d__23.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetException (System.Object exceptionObject) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetException (System.Exception exception) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<InnerRead>d__25.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<InnerRead>d__66.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) (at <00000000000000000000000000000000>:0)
System.AsyncCallback.Invoke (System.IAsyncResult ar) (at <00000000000000000000000000000000>:0)
System.Threading.WaitCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ThreadPoolWorkQueue.Dispatch () (at <00000000000000000000000000000000>:0)
Rethrow as AuthenticationException: A call to SSPI failed, see inner exception.
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<StartOperation>d__58.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<StartOperation>d__23.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetException (System.Object exceptionObject) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetException (System.Exception exception) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<ProcessOperation>d__24.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.AsyncProtocolRequest+<InnerRead>d__25.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[TResult].SetResult (TResult result) (at <00000000000000000000000000000000>:0)
Mono.Net.Security.MobileAuthenticatedStream+<InnerRead>d__66.MoveNext () (at <00000000000000000000000000000000>:0)
System.Threading.ContextCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <00000000000000000000000000000000>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run () (at <00000000000000000000000000000000>:0)
System.Action.Invoke () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction (System.Action action, System.Boolean allowInlining, System.Threading.Tasks.Task& currentTask) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task.FinishContinuations () (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.Task`1[TResult].TrySetResult (TResult result) (at <00000000000000000000000000000000>:0)
System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) (at <00000000000000000000000000000000>:0)
System.AsyncCallback.Invoke (System.IAsyncResult ar) (at <00000000000000000000000000000000>:0)
System.Threading.WaitCallback.Invoke (System.Object state) (at <00000000000000000000000000000000>:0)
System.Threading.ThreadPoolWorkQueue.Dispatch () (at <00000000000000000000000000000000>:0)
Steps to reproduce
Minimal yet complete reproducer code (or URL to code)
@Override
public void initChannel(SocketChannel channel) throws Exception {
final ChannelPipeline pipeline = channel.pipeline();
pipeline.addLast(new IdleStateHandler(MAX_READ_IDLE_SECONDS, 0, 0));
pipeline.addLast(new ProxyMessageDecoder(trustedIpMatcher)); // HAProxy proxy protocol message decoder
final SSLEngine sslEngine = sslContext.newEngine(channel.alloc());
sslEngine.setEnabledProtocols(ENABLED_PROTOCOLS); // "TLSv1.2", client is also configured to use TLS12
pipeline.addLast(null, INNER_SSL_HANDLER, new SslHandler(sslEngine, false, sslForkJoinPool)); // StartTLS false, delegatedTaskExecutor supplied
pipeline.addLast(EvaServerDecoder.class.getSimpleName(), new EvaServerDecoder()); // LengthFieldBasedFrameDecoder
pipeline.addLast(lengthFieldPrepender);
pipeline.addLast(tcpStatisticsHandler); // mbean counters
pipeline.addLast(proxyChannelHandler); // handler to broadcast / disconnect all clients
pipeline.addLast(new ClientSession(beanFactory, channel));
}
Netty version
4.1.70.Final + tcnative 2.0.45.Final fail 4.1.67.Final + tcnative 2.0.40.Final fail 4.1.51.Final + tcnative 2.0.34.Final ok
JVM version (e.g. java -version)
openjdk version “1.8.0_302” OpenJDK Runtime Environment (build 1.8.0_302-b08) OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
OS version (e.g. uname -a)
NAME=“Amazon Linux” VERSION=“2” ID=“amzn” ID_LIKE=“centos rhel fedora” VERSION_ID=“2” PRETTY_NAME=“Amazon Linux 2”
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 60 (60 by maintainers)
@normanmaurer thanks 👌 hit me up if you’re in Italy.
@johnou thanks a lot for all your help to track this down.
ok after that please also try to cherry-pick the fix and see what happens when you put it back
Can you only revert https://github.com/netty/netty/commit/4e86f768b916a34d3a929251eff8428ef47be65d and see what happens ?
haha same here 😃
@johnou keep me posted
@normanmaurer sorry I took a few days off last week, working on a reproducer now. Minimal server extracted, currently plumbing the C# client side.