runtime: InvalidProgramException with message "JIT Compiler encountered an internal limitation." thrown during HTTP calls
This morning when a new EC2 instance was launched for an ASP.NET Core 2.2.5 application of ours in a production environment, all outbound HTTP requests to dependent services failed with the following exception:
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to write data to the transport connection: JIT Compiler encountered an internal limitation.. ---> System.InvalidProgramException: JIT Compiler encountered an internal limitation.
at T System.Threading.LazyInitializer.EnsureInitialized<T>(ref T target)
at ValueTask System.Net.Sockets.Socket.SendAsyncForNetworkStream(ReadOnlyMemory<byte> buffer, SocketFlags socketFlags, CancellationToken cancellationToken)
at ValueTask System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at ValueTask System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory<byte> buffer, CancellationToken cancellationToken)
at ValueTask System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory<byte> source)
at ValueTask System.Net.Http.HttpConnection.FlushAsync()
at async Task<HttpResponseMessage> System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at async Task<HttpResponseMessage> System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at async Task<HttpResponseMessage> System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, bool doRequestAuth, CancellationToken cancellationToken)
at async Task<HttpResponseMessage> System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, bool doRequestAuth, CancellationToken cancellationToken)
at async Task<HttpResponseMessage> System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at async Task<HttpResponseMessage> System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at async Task<HttpResponseMessage> Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
<snip>
The inner-most exception appears to indicate a JIT issue.
This is the first time we’ve ever seen this issue, but it seemed to be unrecoverable. Exceptions were thrown for all HTTP requests made by the application until it was removed from our application load balancer and stopped receiving user traffic.
We’ve captured two full dumps of the dotnet.exe from the instance before it was terminated which we can share with your privately if it helps diagnose the issue.
Let me know if there’s any further information I can share with you other than the dump files.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 16 (16 by maintainers)
Got it, thanks.
First impression is that the dump doesn’t have enough information to pinpoint what went wrong, as the process is past the point of exception. Looks like the issue happened while jitting
LazyInitializer.EnsureInitialized<T>or perhaps the...Corehelper; from what I can tell neither of these was modified by the profiler.I’ll see if I can get somebody else to take a look; perhaps they can spot something I overlooked.
@martincostello apologies for the slow response. Can you open an issue for this on the developer community site?
Once it’s open, you can add a “private” comment and attach your dump file there.