runtime: HttpClient Http/2 doesn't flush frames when hitting the window limit

@deathchurch The current F5 behavior will result in poor performance. The client will send up to the window size, then it will need to wait until it gets a window update from the server. This will result in a stall in sending approx equal to the round trip time.

To improve performance, they should send window updates more frequently. Exactly how frequently is a somewhat complex question. For HttpClient, we send a window update for every approx (total window size / 8) bytes consumed by the app. I believe ASP.NET does something similar. But we’re also looking at changing our receive window size heuristics for 6.0. There’s not necessarily a perfect answer here, but what they are currently doing is definitely suboptimal.

Thanks @geoffkizer I think F5 were of the same opinion, they quoted https://datatracker.ietf.org/doc/html/rfc7540#section-5.2.1, and that on the client side they wait for the window to be filled before sending a window update, also they queried why they would they be expected to update the window if its not being used. I think they were open to any suggestions from Microsoft as to what they could improve.

I see you have a PR, hopefully that will resolve the problem, would that be targeted only to net6.0 or would that be available for net5.0 and netcore3.1?

Curl trace: tcp.stream eq 0

• F5 sets its connection window size to 32768
• The client sets its window size to 33554432(?)
• The client sets sends a connection window update to 33488897
• POST Content-Length 46486 (smaller than HttpClient?)
• DATA 16384
• DATA 16384
• F5 sends a stream window update of 32768
• DATA 13718
• 200 OK response

So F5 is sending a window update in this scenario, which allows the client to continue the request.

Maybe that HttpClient 9s pause is the root issue here, it’s failing to flush that data frame for some reason and F5 doesn’t send a window update until the window is full.

Using .NET 5.0 I was able to reproduce this with Kestrel using a larger request body (100kb) to exceed kestrel’s 96kb stream window size. I made the server avoid consuming the data and sending window updates, an the client times out.

        static async Task Main(string[] args)
        {
            Console.WriteLine("Hello World!");
            using var client = new HttpClient();
            client.Timeout = TimeSpan.FromSeconds(10);
            client.DefaultRequestVersion = HttpVersion.Version20;
            var response = await client.PostAsync("https://localhost:5001", new ByteArrayContent(new byte[100 * 1024]));
            Console.WriteLine(response);
        }

                endpoints.Map("/", async context =>
                {
                    await Task.Delay(TimeSpan.FromSeconds(15));
                    await context.Response.WriteAsync("Hello World!");
                });

trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received HEADERS frame for stream ID 1 with length 28 and flags END_HEADERS
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received DATA frame for stream ID 1 with length 16384 and flags NONE
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received DATA frame for stream ID 1 with length 16384 and flags NONE
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received DATA frame for stream ID 1 with length 16384 and flags NONE
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received DATA frame for stream ID 1 with length 16384 and flags NONE
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received DATA frame for stream ID 1 with length 16384 and flags NONE
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 POST https://localhost:5001/  102400
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint '/'
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received DATA frame for stream ID 1 with length 16384 and flags NONE
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HM8HE4K0ED0D" received RST_STREAM frame for stream ID 1 with length 4 and flags 0x0
trce: Microsoft.AspNetCore.Server.Kestrel[37]

The client sends 5 full frames, 80kb, and then waits for the timeout. Then it sends one more 16kb frame and the RST_STREAM together. 16kb would have put it right at the 96kb limit, but it still should have sent the data. It sounds like it doesn’t flush if it hits the window limit.

I’m going to transfer this over to the HttpClient owners.