runtime: configurable HTTP/2 PING timeouts in HttpClient

Justification

Long-running but idle connections (which happen in gRPC and long poll scenarios) can result in firewalls/etc. dropping connections out from under us, without any reset notification. This causes long timeouts on a client that is only reading data.

These APIs will have HttpClient periodically send ping frames over HTTP/2, which would prevent firewalls from seeing the connection as idle and detect otherwise broken connections sooner than we can right now.

API Proposal

public class SocketsHttpHandler
{
    // if we haven't received data on a connection in X time, send a PING frame.
    public TimeSpan KeepAlivePingDelay { get; set; }

    // if server does not respond to the PING in X time, kill the connection on our side.
    public TimeSpan KeepAlivePingTimeout { get; set; }

    // ****new since last review****
    public HttpKeepAlivePingPolicy KeepAlivePingPolicy { get; set; }
}

// Consider using a bool instead. Proposing enum for better extensibility in future.
public enum HttpKeepAlivePingPolicy
{
    // Only send pings if there are active requests.
    WithActiveRequests,

    // Send pings even for idle connections.
    // Common gRPC servers have behavior that will kill connections that use this. ¯\_(ツ)_/¯
    Always
}

Original issue below

I’ve been evaluating the new gRPC-dotnet and particularly its streaming feature in a long lived connected scenario. I needed the HTTP/2 PING feature to detect the zombie connections (basically to know: “Am I still having an up and running connection to the client?”) on both server and client side. It seems that this feature won’t be implement-able on the client side as long as there’s no support of an API in the HttpClient, as the client part of gRPC-dotnet relies on HttpClient.

It seems that HttpClient already responds to ping but can not send any on demand. Am I right?

So I would like to request a new method in HttpClient class to support sending Http/2 pings or even better a kind of keep alive ping timeout to do it automatically in case there’s no traffic…

BTW: I have also created an issue for the same feature for kestrel (I don’t know what kestrel is using underneath).

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 54 (43 by maintainers)

Most upvoted comments

What I propose be added to SocketsHttpHandler:

public class SocketsHttpHandler
{
    public TimeSpan KeepAliveInterval { get; set; }
    public TimeSpan KeepAliveTimeout { get; set; }
    public bool KeepAliveWithoutRequests { get; set; }
}

KeepAliveInterval controls a timer that sends connection-level HTTP/2 pings to the server. Would start after interval of inactivity in reading incoming data. Has a minimum value of 10 seconds (avoid angering the server with frequent pings). Defaults to 0 (off).

KeepAliveTimeout controls how long the client waits for the ping reply. If the reply is not received in that time then the connection is closed. Defaults to 20 seconds.

KeepAliveWithoutRequests controls whether keep alive pings are sent if there are no active requests. This is useful if someone wants keep alive pings sent only when there are requests in-progress, such as a long running gRPC streaming request. If there are no active requests then keep alive pings would stop and the connection could be closed. Defaults to false.

These are the most important settings from how gRPC clients generally handle keep alive: https://github.com/grpc/grpc/blob/master/doc/keepalive.md. There are additional settings for finer grain control of pings (some are server specific). I don’t believe they are needed.

JamesNK on May 29, 2020

Marking it for 5.0 for now due to raising demand …

karelz on May 28, 2020

It seems the setting Always is undesirable.

Yes it is desirable. I explained why it is desirable here - https://github.com/dotnet/runtime/issues/31198#issuecomment-635622175

To go into an example:

Imagine a client that infrequently sends bursts of data, e.g. 1000 HTTP requests once every hour. Without an option to send keep alive pings - even though there are no active calls - then a load balancer/proxy closes the connection for inactivity. Every time that client wakes up to send the 1000 HTTP requests, all have high latency while they wait for TCP connections to be created, TLS to be established, HTTP/2 connection to be established, and calls to be made.

Customers - including Azure 1st party (I don’t know if I can mention names publicly) - have asked for this feature. They have observed that the scenario impacts their ~P90~ P50 latency numbers.

@chwarr Also goes into detail about why KeepAliveWithoutRequests = true is useful here.

This isn’t gRPC specific. Anyone using HTTP/2 could use this.

We’re going around in circles. Could you please invite me to discuss this feature.

JamesNK on Jul 28, 2020

Naively, I would expect that Ping does not count as “normal” request and does not prolong idle timeout. User should set longer [PooledConnectionIdleTimeout](https://apisof.net/catalog/System.Net.Http.SocketsHttpHandler.PooledConnectionIdleTimeout) to keep idle connections around longer. That is my naive expectations without looking at code - @aik-jahoda can you please confirm?

karelz on Jul 24, 2020

Configurable keep alive pings with Kestrel done - https://github.com/dotnet/aspnetcore/pull/22565

JamesNK on Jun 16, 2020

As an concrete example of HTTP2 PING frames, consider gRPC–which was the impetus for @chrisdot opening this issue.

The gRPC HTTP2 protocol explicitly uses PING frames:

Both clients and servers can send a PING frame that the peer must respond to by precisely echoing what they received. This is used to assert that the connection is still live as well as providing a means to estimate end-to-end latency. If a server initiated PING does not receive a response within the deadline expected by the runtime all outstanding calls on the server will be closed with a CANCELLED status. An expired client initiated PING will cause all calls to be closed with an UNAVAILABLE status. Note that the frequency of PINGs is highly dependent on the network environment, implementations are free to adjust PING frequency based on network and application requirements.

The https://github.com/grpc/grpc-dotnet/ project is building a gRPC implementation completely in .NET. One of its goals is to be able to interoperate with the other gRPC implementations.

The C core library implementation of gRPC implements the PING part of the protocol and calls it keepalive. The Go implementation is similar.

chwarr on Oct 18, 2019

But now I wonder how will respond to client http2 pings on the server? We use Kestrel there.

This is where my other related issue comes in…

chrisdot on May 13, 2020