runtime: Sudden OutOfMemoryExceptions with 2.1.1-runtime-alpine

From @natethegreat44 on June 25, 2018 17:40

Hi,

We have an ASP.NET core app that recently started receiving unexplainable OutOfMemoryExceptions after deployment on Thursday, 21 June, 2018. The exceptions are not usually fatal, but once they start, they grow worse until eventually the dotnet process crashes. Things are usually good for about 18-24 hours after launch, and it doesn’t seem to matter much whether things are under load or not - we’re using blue/green deployments, and our inactive cluster experiences crashes about as much as the active cluster.

This app serves as an integration layer for other RESTful HTTP services and as such makes extensive use of the HttpClient (i.e. we’re not explicitly using any new HTTP Apis from core 2.1).

We’re using the standard Alpine Docker images, running on Linux in AWS. Prior to 21 June, we were on 2.1.0 (aka 2.1.300) since 13 June, and 2.0 for quite a while before that, without any OutOfMemoryExceptions. Apparently our build on Thursday was upgraded to 2.1.1 via the 2.1-sdk-alpine and 2.1-runtime-alpine docker base images.

Using tools like ps, htop, it does not appear that memory is actually the cause here - usage typically sits at about 1GB out of 16GB available.

As such, it really looks like something in 2.1.1 is conflicting with our code. An example stack trace is below.

We did manage to capture a few core dumps that we’re trying to dig into. In the mean time, if there are a are any other details that might be helpful, please let us know.

Thank you, Nate

[2018-06-24T22:35:24.3939995Z] Level=Error, Category=Whibbidy.Common.Web.HttpHelper, Id=SYSTEM, Message=Error when making web request to GET : Url=https://<redacted>. DoRetry=False.  Type=System.Net.Http.HttpRequestException Exception=System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Runtime.InteropServices.Marshal.GetFunctionPointerForDelegateInternal(Delegate d)
   at Interop.Crypto.X509StoreCtxSetVerifyCallback(SafeX509StoreCtxHandle ctx, X509StoreVerifyCallback callback)
   at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.BuildChain(X509Certificate2 leaf, HashSet`1 candidates, HashSet`1 systemTrusted, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan& remainingDownloadTime)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at System.Net.Security.CertificateValidation.BuildChainAndVerifyProperties(X509Chain chain, X509Certificate2 remoteCertificate, Boolean checkCertName, String hostName)
   at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken)
   at System.Net.Security.SslState.CompleteHandshake(ProtocolToken& alertToken)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__47_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Whibbidy.Common.Web.HttpHelper.ProcessRequest(String uri, HttpMethod method, HttpHeaders headers, Int32 timeout, String body, String[] redact, Boolean doRetry) in /Whibbidy/src/Whibbidy.Common/Web/HttpHelper.cs:line 259    at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.WaitForCreatedConnectionAsync(ValueTask`1 creationTask)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Whibbidy.Common.Web.HttpHelper.ProcessRequest(String uri, HttpMethod method, HttpHeaders headers, Int32 timeout, String body, String[] redact, Boolean doRetry) in /Whibbidy/src/Whibbidy.Common/Web/HttpHelper.cs:line 259

Copied from original issue: dotnet/coreclr#18637