runtime: The server returned an invalid or unrecognized response" error when using System.Net.Http.HttpClient

Getting this bug into the right place: https://github.com/dotnet/core/issues/321

I am facing a problem when making concurrent requests to a web api when using dotnet core. Sometimes i’m getting the below error; the error is not happening all the time (usually it happens once or twice in 600 concurrent calls).

System.Net.Http.WinHttpException: The server returned an invalid or unrecognized response  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

Here’s the test method I use for testing:

    [Fact]
    public void PostToStardogParallelTaskWithSendAsync()
    {
        int x = 600;
        Task[] allTasks = new Task[x];

        using (var httpClient = new HttpClient())
        {
            var byteArray = Encoding.ASCII.GetBytes($"user:password");                

            for (int k = 0; k < 1; k++)
            {
                for (int i = 0; i < x; i++)
                {
                    System.Diagnostics.Debug.WriteLine($"Starting tsk {i}...");
                    allTasks[i] = Task.Factory.StartNew(async (ii) =>
                    {
                        var d = DateTime.Now;
                        var query = $"INSERT DATA {{ <urn:subj> <urn:pred> <urn:obj>. }}";
                        var content = new StringContent(query, Encoding.UTF8, "application/sparql-update");

                        HttpRequestMessage rm = new HttpRequestMessage(HttpMethod.Post, "MyStardogWebApiUrl");
                        rm.Headers.TryAddWithoutValidation("accept", "text/boolean");
                        rm.Headers.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
                        rm.Content = content;                            
                        try
                        {
                            var response = await httpClient.SendAsync(rm);
                        }
                        catch (Exception e)
                        {
                            System.Diagnostics.Debug.WriteLine($"\nThe following exception have been thrown: {e}");                            
                        }

                        System.Diagnostics.Debug.WriteLine($"Task {ii} ended in: {(DateTime.Now - d).TotalMilliseconds}.");

                    }, i).Unwrap();
                }
                try
                {
                    Task.WaitAll(allTasks);
                }
                catch (Exception e)
                {
                    Console.WriteLine($"nThe following exception have been thrown: {e}");                       
                }
            }
        }
    }

When this error occurs, the response looks like this: {StatusCode: 200, ReasonPhrase: ‘OK’, Version: 1.1, Content: <null>, Headers:{}}

Calling the same method from a testing project that uses dotnet framework 4.5 does not cause any issue.

Could you please advise? Thanks!

Corina

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 30 (18 by maintainers)

Most upvoted comments

We have finished the analysis of this issue. The problem is due to the server violating the RFC by returning an entity-body. It is sending a 0-byte chunk. The response to a HEAD request should be to send absolutely no bytes of data for the response body.

The reason for the difference in behavior between .NET Framework and .NET Core is due to race conditions and architectural differences.

.NET Framework will fail right away after the first request. It fails, by-design, with a ProtocolViolationException.

.NET Core (on Windows) uses native WinHTTP. This is the explanation from the WinHTTP team about why it is failing later on (due to race condition).

The server is sending entity-body in response to a HEAD request which is illegal. This results in a race between a request picking up the connection and the connection being aborted for having unexpected data. To be clear the illegal body is the 0 length chunk. There should be nothing after the headers in response to a HEAD. The WinHTTP stack was initially built to support pipelining, so any data after the headers is processed after the current request since by RFC this data is not part of the current response. This means the current request will complete, and then the subsequent data will be processed asynchronously. When it is processed there is no active request, so the connection is terminated. This races with the connection being picked up out of the pool.

We are closing this issue. Please open up a new issue with the ASP.NET team regarding WebListener.

davidsh on Apr 4, 2017

Actually, after looking at your code again, you are creating a new HttpRequestMessage each time! So, this is very strange because that exception should not be returned.

I ran the repro again and was able to generate the exception we’re investigating.

The exception happened after 906 iterations.

We will investigate. Thx.

davidsh on Apr 1, 2017

Although the root cause is on the server side, IMHO the clients should consistently report that the server returned a malformed HTTP 1.1 response. I’m interested to understand whether behavior differences are considered bugs or not.

We don’t consider this a bug. The difference is due to architectural differences and race conditions in the platform OS APIs that are used for HTTP. As indicated above, the server is violating the RFC and in those cases, some behaviors of the client can be undefined or variable in these edge cases.

@davidsh: I don’t think we’ve confirmed that the original issue reported by @corinas and @Petermarcu is actually caused by the same issue, so we may want to wait for their input?

If there an actionable repro that demonstrates this same error but without involving a HEAD request with invalid entity-body response content, please open up a new issue. It will make things easier for tracking.

davidsh on Apr 5, 2017

I’ve written a test client for full framework, see https://github.com/devatwork/ReproduceUnrecognizedResponse. Interestingly enough full framework also thinks this is an issue on the server side as well. Stack trace:

System.Net.WebException: The server committed a protocol violation. Section=ResponseStatusLine
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)

This seems to suggest that the server application returns an invalid response. In both Jos’s case and my case the server application is using WebListener/HttpSysServer. The HTTP response messages are identical according to a Fiddler trace I’ve run.

devatwork on Apr 3, 2017

@JosVerburg Please put together a complete, simple, repro with a Visual Studio 2015 or 2017 solution. You can simply zip up the solution and post it here. That would be the best way for us to investigate.

davidsh on Mar 29, 2017