runtime: .NET Core 2.2 - HttpClient throws IOException: The server returned an invalid or unrecognized response.

This occurs to be happening on Linux (docker) and OSX on both .NET Core 2.1 and 2.2 (latest), I’ve heard it’s also reproducible on windows but I’ve yet to confirm… We have many jobs that talk to Elasticsearch and it’s only this one specific one that occasionally blows up. I’ve brought in all the dependency source code minus the CoreCLR and have been able to get it under the debugger:

 System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The server returned an invalid or unrecognized response.
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Elasticsearch.Net.HttpConnection.RequestAsync[TReturn](RequestData requestData, CancellationToken cancellationToken)

The last line in the stack trace points to here: https://github.com/elastic/elasticsearch-net/blob/5.x/src/Elasticsearch.Net/Connection/HttpConnection-CoreFx.cs#L120

I’ve inspected the http client instance under the debugger and it looks completely normal. If I reexecute the request when it fails it works. I’ve enabled all the logging I can (verbose) and nothing shows up… I’ve created a small sample that should be making the same request but isn’t blowing up… I’m in the process of creating a better sample that uses the nest client to see if that reproduces this. If you have any ideas on how I can get better logging / inspect the raw request as Wireshark and Charles isn’t helping any :.

using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApp1
{
    class Program
    {
        private const string JSON = "{\"size\":801,\"query\":{\"bool\":{\"filter\":[{\"bool\":{\"must\":[{\"range\":{\"metaData.nextRunDateUtc\":{\"gte\":\"0001-01-01T05:51:00Z\",\"lte\":\"2019-01-29T23:45:55.436645Z\"}}},{\"term\":{\"isEnabled\":{\"value\":true}}}]}}]}}}";

        public static async Task Main(string[] args)
        {
            Console.WriteLine("Hello World!");

            var client = new HttpClient { BaseAddress = new Uri("http://localhost:9200")};
            var counter = 1;
            while (true)
            {
                try
                {
                    var response = await client.SendAsync(CreateRequestMessage());
                    Console.WriteLine($"{counter++}: {response.StatusCode}");
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex);
                }
            }
        }
        
        protected static HttpRequestMessage CreateRequestMessage()
        {
            var requestMessage = new HttpRequestMessage(System.Net.Http.HttpMethod.Post, "/test/test/_search?ignore_unavailable=true");
            requestMessage.Headers.Connection.Clear();
            requestMessage.Headers.ConnectionClose = false;
            requestMessage.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            requestMessage.Content = new StreamContent(new MemoryStream(Encoding.UTF8.GetBytes(JSON)));
            requestMessage.Content.Headers.ContentType = new MediaTypeHeaderValue("application/json");

            return requestMessage;
        }
    }
}

I’ve also looked through some of the other issues here: https://github.com/dotnet/corefx/issues?utf8=✓&q=is%3Aissue+“The+server+returned+an+invalid+or+unrecognized+response”