extensions: Shared circuit breaker when registering a resilience handler with ConfigureHttpClientDefaults

Description

When registering the standard resilience handler using builder.Services.ConfigureHttpClientDefaults(x => x.AddStandardResilienceHandler), all HttpClients seem to be using a shared circuit breaker. When one client is causing the circuit to open, the circuit opens for all clients.

Reproduction Steps

Here is my project file:

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Http.Resilience" Version="8.2.0" />
  </ItemGroup>

</Project>

Here is my appsettings.json (modified to be able to more easily reproduce the issue):

{
  "RetryOptions": {
    "Retry": {
      "Delay": "00:00:00",
      "MaxRetryAttempts": 1
    }
  }
}

And here is the Program.cs:

var builder = WebApplication.CreateBuilder(args);

var section = builder.Configuration.GetSection("RetryOptions");

var succeedingClientBuilder = builder.Services.AddHttpClient<SucceedingClient>(x =>
    x.BaseAddress = new Uri("https://ipv4.icanhazip.com/"));

var failingClientBuilder = builder.Services.AddHttpClient<FailingClient>(x =>
    x.BaseAddress = new Uri("http://the-internet.herokuapp.com/status_codes/500"));

var useHttpClientDefaults = true;
if (useHttpClientDefaults)
{
    builder.Services.ConfigureHttpClientDefaults(x => x.AddStandardResilienceHandler(section));
}
else
{
    succeedingClientBuilder.AddStandardResilienceHandler(section);
    failingClientBuilder.AddStandardResilienceHandler(section);
}

var app = builder.Build();

app.MapGet("/test", async (SucceedingClient succeedingClient, FailingClient failingClient) =>
{
    var succeedingClientResult = await succeedingClient.Get();
    var failingClientResult = await failingClient.Get();

    return $"""
        SucceedingClient result: {succeedingClientResult}
        FailingClient result: {failingClientResult}
        """;
});

app.Run();

public class SucceedingClient(HttpClient client)
{
    public async Task<string> Get()
    {
        try
        {
            using var response = await client.GetAsync("");
            return $"Status code {response.StatusCode}";
        }
        catch (Exception e)
        {
            return $"Exception {e.Message}";
        }
    }
}

public class FailingClient(HttpClient client)
{
    public async Task<string> Get()
    {
        try
        {
            using var response = await client.GetAsync("");
            return $"Status code {response.StatusCode}";
        }
        catch (Exception e)
        {
            return $"Exception {e.Message}";
        }
    }
}

Run this sample code and continuously hit the /test endpoint until the circuit breaker kicks in.

Expected behavior

I would expect the circuit would only open for the client that is facing issues. From the reproduction steps, the circuit should not open for SucceedingClient when it opens for FailingClient.

Actual behavior

The circuit breaker opens for both the SucceedingClient and the FailingClient, even though only FailingClient is receiving status code 500.

Regression?

No response

Known Workarounds

Use AddStandardResilienceHandler on each HttpClient instead of using ConfigureHttpClientDefaults.

Configuration

dotnet --info .NET SDK: Version: 8.0.200 Commit: 438cab6a9d Workload version: 8.0.200-manifests.e575128c

Runtime Environment: OS Name: Windows OS Version: 10.0.22631 OS Platform: Windows RID: win-x64 Base Path: C:\Program Files\dotnet\sdk\8.0.200\

.NET workloads installed: There are no installed workloads to display.

Host: Version: 8.0.2 Architecture: x64 Commit: 1381d5ebd2

.NET SDKs installed: 8.0.200 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: Not found

Other information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 18 (16 by maintainers)

Most upvoted comments

Regarding the enhancement that will allow to override the Standard Resilience Handler.

As @martintmk pointed out we can implement an improvement when a second (and any following) attempt to register the Standard Resilience Handler doesn’t add a new handler, but changes the configuration of the first registered handler. If we implement that then the following code sample:

var services = new ServiceCollection();
services.ConfigureHttpClientDefaults((IHttpClientBuilder x) => x.AddStandardResilienceHandler());

services.AddHttpClient("FailingClient", x => x.BaseAddress = new Uri("https://failing.api"));

services.AddHttpClient("SucceedingClient", x => x.BaseAddress = new Uri("https://succeeding.api/"))
    AddStandardResilienceHandler();

will result in having two HttpClients:

  • “FailingClient” uses the default standard resilience handler. It will share the default handler with any other HttpClient that doesn’t explicitly register its own resilience handler.
  • “SucceedingClient” will have its own resilience handler. Its handler will replace the default handler. Failures caused by the “FailingClient” will not affect the resilience handler of the “SucceedingClient”.

And this solves the issue with the “shared circuit breaker”. The downside is that a customer has to explicitly register a named client and register the standard resilience for it.

But there still will be issues we’ll need to address. They are listed below.

1. StandardResilienceHandler vs StandardHedgingHandler

As @martintmk said, having two standard resilience handlers in HttpClient would break it. Similarly, having two standard hedging handlers or combination of hedging and resilience handlers will break HttpClient. With the approach/solution proposed above we will need not only overwrite the configuration of the handler but also its type, e.g. if someone registers standard resilience handler and then standard hedging handler, then he/she will get HttpClient with the only standard hedging handler, i.e. the last registered handler will “win”.

2. What if we need to change the order of the resilience handler in HttpClient pipeline?

With the proposed approach it is not achievable. We can consider introducing a method RemoveAllStandardResilienceHandlers similar to RemoveAllLoggers. It will help to remove all standard resilience handlers and put your resilience handler in the required place in the request pipeline. But since there is no such need at the moment we can get back to this when the need occurs.

3. Resilience handlers could be registered by 3rd party libraries

If some 3rd party library registers its own resilience handler then with the the last registered resilience handler “wins” approach we’ll get HttpClient with the resilience handler provided by 3rd party’s library which could be undesirable. Do we need to introduce a special method with TryAdd* semantics that will not overwrite the already registered handler? We’ll need to address this too, but I think we can skip for now this since there is no such a request from customers.


To conclude, I suggest to implement the approach suggested above: the last registered resilience or hedging handler overwrites settings of the first registered handler. We don’t allow more than one standard resilience or hedging handler in HttpClient. We’ll postpone solving potential issues 2 and 3 mentioned above until customers come to us with corresponding asks.

So, what do you think?

We could also close #4818 as dup of this one, and continue here 😃

To sum up, we have two issues:

  1. A documentation gap which will address the immediate issue (more or less a tactical fix).
  2. A possible enhancement that will allow overriding the HTTP client resilience settings.

@iliar-turdushev , Great investigation!

services.ConfigureHttpClientDefaults(x => x x.AddStandardResilienceHandler().SelectPipelineByAuthority());

This helps slightly in a way that at least multiple clients with different authorities won’t share the same circuit breaker. (which is quite an improvement)

However, consider this example:

var services = new ServiceCollection();
services.AddLogging(builder => builder.AddConsole().SetMinimumLevel(LogLevel.Debug));
services.ConfigureHttpClientDefaults(builder =>
{
    builder.AddStandardResilienceHandler().SelectPipelineByAuthority();

});

var factory = services.BuildServiceProvider().GetRequiredService<IHttpClientFactory>();

Console.WriteLine("client-1");
var client1 = factory.CreateClient("client-1");
await client1.GetStringAsync("https://jsonplaceholder.typicode.com/todos/1");

Console.WriteLine("client-2");
var client2 = factory.CreateClient("client-2");
await client2.GetStringAsync("https://jsonplaceholder.typicode.com/todos/1");

That output’s the following:

info: System.Net.Http.HttpClient.client-1.ClientHandler[100]
      Sending HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.client-1.ClientHandler[101]
      Received HTTP response headers after 158.4074ms - 200
info: Polly[3]
      Execution attempt. Source: '-standard/https://jsonplaceholder.typicode.com/Standard-Retry', Operation Key: '', Result: '200', Handled: 'False', Attempt: '0', Execution Time: 168.3799ms
dbug: Polly[2]
      Resilience pipeline executed. Source: '-standard/https://jsonplaceholder.typicode.com', Operation Key: '', Result: '200', Execution Time: 180.2802ms
info: System.Net.Http.HttpClient.client-1.LogicalHandler[101]
      End processing HTTP request after 227.9248ms - 200
client-2
info: System.Net.Http.HttpClient.client-2.LogicalHandler[100]
      Start processing HTTP request GET https://jsonplaceholder.typicode.com/todos/1
dbug: Polly[1]
      Resilience pipeline executing. Source: '-standard/https://jsonplaceholder.typicode.com', Operation Key: '(null)'
info: System.Net.Http.HttpClient.client-2.ClientHandler[100]
      Sending HTTP request GET https://jsonplaceholder.typicode.com/todos/1
info: System.Net.Http.HttpClient.client-2.ClientHandler[101]
      Received HTTP response headers after 102.2284ms - 200
info: Polly[3]
      Execution attempt. Source: '-standard/https://jsonplaceholder.typicode.com/Standard-Retry', Operation Key: '', Result: '200', Handled: 'False', Attempt: '0', Execution Time: 102.366ms
dbug: Polly[2]
      Resilience pipeline executed. Source: '-standard/https://jsonplaceholder.typicode.com', Operation Key: '', Result: '200', Execution Time: 102.4594ms
info: System.Net.Http.HttpClient.client-2.LogicalHandler[101]
      End processing HTTP request after 102.6431ms - 200

Notice the -standard/https://jsonplaceholder.typicode.com/Standard-Retry which completely omits the client-1 which should be there as identifier of the pipeline. This is because the ConfigureHttpClientDefaults does not use any client name.

Also @RussKie, your example bellow breaks the pipeline as it adds the additional standard resilience handler. I don’t believe there is a use-case where 2 standard pipelines are valid on a single HttpClient.

var builder = WebApplication.CreateBuilder(args);

var section = builder.Configuration.GetSection("RetryOptions");

builder.Services.ConfigureHttpClientDefaults(x => x.AddStandardResilienceHandler(section));

var succeedingClientBuilder = builder.Services.AddHttpClient<SucceedingClient>(x => x.BaseAddress = new Uri("https://ipv4.icanhazip.com/"))4
                                           .AddStandardResilienceHandler();
var failingClientBuilder = builder.Services.AddHttpClient<FailingClient>(x => x.BaseAddress = new Uri("http://the-internet.herokuapp.com/status_codes/500"))
                                           .AddStandardResilienceHandler();

IMHO, much better behavior would be:

  • ConfigureHttpClientDefaults + AddStandardResilienceHandler properly preserves the http client name in the telemetry. As additional benefit, there is no need to call SelectPipelineByAuthority. However, I don’t think this is doable given the limitations of ConfigureHttpClientDefaults .
  • ConfigureHttpClientDefaults + AddStandardResilienceHandler + explicit AddStandardResilienceHandler. This call won’t actually add a new handler. Instead, it preserves the order of the original handler and just allows changing the configuration for it. Essentially, duplicates won’t be allowed for standard pipeline.

The issue occurs because ConfigureHttpClientDefaults(x => x.AddStandardResilienceHandler()) registers a single resilience pipeline with a name standard that is then shared by both SucceedingClient and FailingClient. As a result, they both share the circuit breaker of the standard pipeline.

When we use two separate statements to register the resilience pipeline

succeedingClientBuilder.AddStandardResilienceHandler(section)
failingClientBuilder.AddStandardResilienceHandler(section)

we create two distinct resilience pipelines named SucceedingClient-standard and FailingClient-standard, and each of them has its own circuit breaker. Each of these pipelines is then used by the respective HttpClients.

The reason for that is how AddStandardResilienceHandler() and ConfigureHttpClientDefaults() methods work. AddStandardResilienceHandler() relies on IHttpClientBuilder.Name when registering and resolving the corresponding resilience pipeline. Whenever you call AddStandardResilienceHandler() you register a resilience pipeline for the given IHttpClientBuilder.Name, and when you create an HttpClient the corresponding resilience pipeline is retrieved by the name or type of the HttpClient, i.e. by the given IHttpClientBuilder.Name. ConfigureHttpClientDefaults() executes its lambda x => x.AddStandardResilienceHandler() only once, and the x argument is an IHttpClienBuilder with the Name=null, i.e. it doesn’t have information about the actual name of HttpClient. It doesn’t execute the lambda for each registered HttpClient. As a result, we have a single resilience pipeline (and hence a single shared circuit breaker) which is applied to all HttpClients regardless of their name or type.

@JamesNK Can ConfigureHttpClientDefaults(Action<IHttpClientBuilder> configure) be updated to provide the actual name/type of the HttpClient being resolved? As I understand, currently its configure action is executed immediately when ConfigureHttpClientDefaults() is invoked, therefore it doesn’t have the actual Name of HttpClient. And to be able to provide the Name it should be updated to execute the configure action when the actual HttpClient is being resolved/created, i.e. when there is information about its type/name. Can we update ConfigureHttpClientDefaults() in a such way?

Workaround

@sveinungf As a workaround, you can configure the standard resilience pipeline to be applied per HTTP request authority:

builder.Services.ConfigureHttpClientDefaults(x => x
    .AddStandardResilienceHandler(section)
    .SelectPipelineByAuthority());

If you do so, you’ll have distinct resilience pipelines per each HTTP request authority. It’ll solve the shared circuit breaker issue. You can even specify a custom criteria how to select resilience pipelines using the method SelectPipelineBy.