runtime: Sporadic OpenSSL decryption fails for HttpClient responses

Description

Hello, we recently began seeing errors reported when doing M2M https requests between our API’s. We can see from the application logs that requests are received, and correctly responded to, but when the initiating API attempts to read the response, it gets an decryption exception. The faults occur randomly, and only a very small percentage of our requests fail. (~0,1%). We see no errors on the receiving api end.

Our setup:

• run on Azure Kubernetes Service 1.22.6.
• recently upgraded from .Net Core 3.1 LTS to .NET 6.0 LTS.
• running these Docker images:
  • mcr.microsoft.com/dotnet/aspnet:6.0-alpine
  • mcr.microsoft.com/dotnet/aspnet:6.0 (Checked both and they are both at .Net 6.0.7)
• We are using multiple authentication schemes, each of them trying to authenticate incoming requests. (AddJwtBearer)

We tried using the 6.0.4 dotnet runtime docker image, but still got faults.

Our issue is very similar to the one reported here: https://github.com/dotnet/runtime/issues/64492

Reproduction Steps

We have not identified reproduction steps yet.

Expected behavior

No errors occuring when using HttpClient to send and receive requests.

Actual behavior

• Api A sends a request to Api B.
• B receives request, and responds without errors.
• A produces either of the following error:

---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The decryption operation failed, see inner exception.
 ---> Interop+OpenSsl+SslException: Decrypt failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamPal.DecryptMessage(SafeDeleteSslContext securityContext, Span`1 buffer, Int32& offset, Int32& count)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

and

 System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The decryption operation failed, see inner exception.
 ---> Interop+OpenSsl+SslException: Decrypt failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:04067084:rsa routines:rsa_ossl_public_decrypt:data too large for modulus
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamPal.DecryptMessage(SafeDeleteSslContext securityContext, Span`1 buffer, Int32& offset, Int32& count)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](TIOAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Regression?

Did not see this problem on .NET Core 3.1 LTS.

Known Workarounds

No response

Configuration

.NET 6.0.7 runtime, AKS linux containers, Alpine and Debian distro.

Other information

Related issue: https://github.com/dotnet/runtime/issues/64492