runtime: SocketsHttpHandler throws exception when authenticating proxy or server closes first 407 response

Found this bug while investigating dotnet/runtime#26397.

Many servers and proxy servers that require authentication will send ‘Connection: close’ (and close the TCP connection) for the first 407 response. This is frequently true for hardware firewalls, etc.

See: https://www.cisco.com/c/en/us/support/docs/security/web-security-appliance/117931-technote-ntml.pdf

8 The proxy FINs this TCP socket. This is correct and normal.

SocketsHttpHandler doesn’t have a problem with this for Basic or Digest schemes. But for Windows auth schemes such as Negotiate or NTLM, it will throw an exception:

System.Net.Http.HttpRequestException: Authentication failed because the connection could not be reused at System.Net.Http.HttpConnection.DrainResponseAsync(HttpResponseMessage response) in s:\GitHub\corefx\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnection.cs

https://github.com/dotnet/corefx/blob/f5c0d3f6e36ffd706b56b67a17699ca4ce0fca00/src/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnection.cs#L1534

It is true that Windows auth schemes require the connection to stay alive during the multi-leg challenge/response between client and server (or proxy). But that is only true once the challenge/response process starts. And that is after the client responds to the first 407 and sends a ‘Proxy-Authorization’ (or ‘Authorization’) header with a based64-encoded blob with the Negotiate or NTLM scheme.

Repro code showing an authenticating proxy and resulting in an exception while trying to connect to a destination HTTP server. Note: this doesn’t repro if the destination server is HTTPS and thus CONNECT tunneling would be used.

static void Main()
{
    Console.WriteLine($"(Framework: {Path.GetDirectoryName(typeof(object).Assembly.Location)})");
    Socket listener = null;

    // Start a "proxy" server in the background.
    listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
    listener.Bind(new IPEndPoint(IPAddress.Loopback, 0));
    listener.Listen(int.MaxValue);
    var ep = (IPEndPoint)listener.LocalEndPoint;
    var proxyUri = new Uri($"http://{ep.Address}:{ep.Port}/");

    Task.Run(async () =>
    {
        while (true)
        {
            Socket s = await listener.AcceptAsync();
            var ignored = Task.Run(() =>
            {
                using (var ns = new NetworkStream(s))
                using (var reader = new StreamReader(ns))
                using (var writer = new StreamWriter(ns) { AutoFlush = true })
                {
                    int request = 1;
                    while (true)
                    {
                        string line = null;
                        while (!string.IsNullOrEmpty(line = reader.ReadLine()))
                        {
                            Console.WriteLine($"    [request:{request}] Server received: {line}");
                        }

                        Console.WriteLine($"    Server sending response\r\n");
                        writer.Write(
                            "HTTP/1.1 407 Proxy Authentication Required\r\n" +
                            "Proxy-Authenticate: NEGOTIATE\r\n" +
                            "Proxy-Authenticate: NTLM\r\n" +
                            "Cache-Control: no-cache\r\n" +
                            "Pragma: no-cache\r\n" +
                            "Proxy-Connection: close\r\n" +
                            "Connection: close\r\n" +
                            "Content-Length: 0\r\n\r\n");
                        request++;
                    }
                }
            });
        }
    });
            
    var serverUri = new Uri("http://corefx-net.cloudapp.net/echo.ashx/");

    var handler = new HttpClientHandler();
    handler.Proxy = new WebProxy(proxyUri);
    handler.Proxy.Credentials = CredentialCache.DefaultCredentials;
    using (var client = new HttpClient(handler))
    {
        Console.WriteLine($"Doing GET for {serverUri}");
        HttpResponseMessage response = client.GetAsync(serverUri).GetAwaiter().GetResult();
    }
}