extensions: Shared circuit breaker when registering a resilience handler with ConfigureHttpClientDefaults
Description
When registering the standard resilience handler using builder.Services.ConfigureHttpClientDefaults(x => x.AddStandardResilienceHandler), all HttpClients seem to be using a shared circuit breaker. When one client is causing the circuit to open, the circuit opens for all clients.
Reproduction Steps
Here is my project file:
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.Http.Resilience" Version="8.2.0" />
</ItemGroup>
</Project>
Here is my appsettings.json (modified to be able to more easily reproduce the issue):
{
"RetryOptions": {
"Retry": {
"Delay": "00:00:00",
"MaxRetryAttempts": 1
}
}
}
And here is the Program.cs:
var builder = WebApplication.CreateBuilder(args);
var section = builder.Configuration.GetSection("RetryOptions");
var succeedingClientBuilder = builder.Services.AddHttpClient<SucceedingClient>(x =>
x.BaseAddress = new Uri("https://ipv4.icanhazip.com/"));
var failingClientBuilder = builder.Services.AddHttpClient<FailingClient>(x =>
x.BaseAddress = new Uri("http://the-internet.herokuapp.com/status_codes/500"));
var useHttpClientDefaults = true;
if (useHttpClientDefaults)
{
builder.Services.ConfigureHttpClientDefaults(x => x.AddStandardResilienceHandler(section));
}
else
{
succeedingClientBuilder.AddStandardResilienceHandler(section);
failingClientBuilder.AddStandardResilienceHandler(section);
}
var app = builder.Build();
app.MapGet("/test", async (SucceedingClient succeedingClient, FailingClient failingClient) =>
{
var succeedingClientResult = await succeedingClient.Get();
var failingClientResult = await failingClient.Get();
return $"""
SucceedingClient result: {succeedingClientResult}
FailingClient result: {failingClientResult}
""";
});
app.Run();
public class SucceedingClient(HttpClient client)
{
public async Task<string> Get()
{
try
{
using var response = await client.GetAsync("");
return $"Status code {response.StatusCode}";
}
catch (Exception e)
{
return $"Exception {e.Message}";
}
}
}
public class FailingClient(HttpClient client)
{
public async Task<string> Get()
{
try
{
using var response = await client.GetAsync("");
return $"Status code {response.StatusCode}";
}
catch (Exception e)
{
return $"Exception {e.Message}";
}
}
}
Run this sample code and continuously hit the /test endpoint until the circuit breaker kicks in.
Expected behavior
I would expect the circuit would only open for the client that is facing issues. From the reproduction steps, the circuit should not open for SucceedingClient when it opens for FailingClient.
Actual behavior
The circuit breaker opens for both the SucceedingClient and the FailingClient, even though only FailingClient is receiving status code 500.
Regression?
No response
Known Workarounds
Use AddStandardResilienceHandler on each HttpClient instead of using ConfigureHttpClientDefaults.
Configuration
dotnet --info
.NET SDK: Version: 8.0.200 Commit: 438cab6a9d Workload version: 8.0.200-manifests.e575128cRuntime Environment: OS Name: Windows OS Version: 10.0.22631 OS Platform: Windows RID: win-x64 Base Path: C:\Program Files\dotnet\sdk\8.0.200\
.NET workloads installed: There are no installed workloads to display.
Host: Version: 8.0.2 Architecture: x64 Commit: 1381d5ebd2
.NET SDKs installed: 8.0.200 [C:\Program Files\dotnet\sdk]
.NET runtimes installed: Microsoft.AspNetCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 3.1.32 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]
Environment variables: Not set
global.json file: Not found
Other information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 18 (16 by maintainers)
Regarding the enhancement that will allow to override the Standard Resilience Handler.
As @martintmk pointed out we can implement an improvement when a second (and any following) attempt to register the Standard Resilience Handler doesn’t add a new handler, but changes the configuration of the first registered handler. If we implement that then the following code sample:
will result in having two
HttpClients:And this solves the issue with the “shared circuit breaker”. The downside is that a customer has to explicitly register a named client and register the standard resilience for it.
But there still will be issues we’ll need to address. They are listed below.
1. StandardResilienceHandler vs StandardHedgingHandler
As @martintmk said, having two standard resilience handlers in HttpClient would break it. Similarly, having two standard hedging handlers or combination of hedging and resilience handlers will break HttpClient. With the approach/solution proposed above we will need not only overwrite the configuration of the handler but also its type, e.g. if someone registers standard resilience handler and then standard hedging handler, then he/she will get HttpClient with the only standard hedging handler, i.e. the last registered handler will “win”.
2. What if we need to change the order of the resilience handler in HttpClient pipeline?
With the proposed approach it is not achievable. We can consider introducing a method
RemoveAllStandardResilienceHandlerssimilar to RemoveAllLoggers. It will help to remove all standard resilience handlers and put your resilience handler in the required place in the request pipeline. But since there is no such need at the moment we can get back to this when the need occurs.3. Resilience handlers could be registered by 3rd party libraries
If some 3rd party library registers its own resilience handler then with the the last registered resilience handler “wins” approach we’ll get HttpClient with the resilience handler provided by 3rd party’s library which could be undesirable. Do we need to introduce a special method with
TryAdd*semantics that will not overwrite the already registered handler? We’ll need to address this too, but I think we can skip for now this since there is no such a request from customers.To conclude, I suggest to implement the approach suggested above: the last registered resilience or hedging handler overwrites settings of the first registered handler. We don’t allow more than one standard resilience or hedging handler in HttpClient. We’ll postpone solving potential issues 2 and 3 mentioned above until customers come to us with corresponding asks.
So, what do you think?
We could also close #4818 as dup of this one, and continue here 😃
To sum up, we have two issues:
@iliar-turdushev , Great investigation!
This helps slightly in a way that at least multiple clients with different authorities won’t share the same circuit breaker. (which is quite an improvement)
However, consider this example:
That output’s the following:
Notice the
-standard/https://jsonplaceholder.typicode.com/Standard-Retrywhich completely omits theclient-1which should be there as identifier of the pipeline. This is because theConfigureHttpClientDefaultsdoes not use any client name.Also @RussKie, your example bellow breaks the pipeline as it adds the additional standard resilience handler. I don’t believe there is a use-case where 2 standard pipelines are valid on a single
HttpClient.IMHO, much better behavior would be:
ConfigureHttpClientDefaults + AddStandardResilienceHandlerproperly preserves the http client name in the telemetry. As additional benefit, there is no need to callSelectPipelineByAuthority. However, I don’t think this is doable given the limitations ofConfigureHttpClientDefaults.ConfigureHttpClientDefaults + AddStandardResilienceHandler+ explicitAddStandardResilienceHandler. This call won’t actually add a new handler. Instead, it preserves the order of the original handler and just allows changing the configuration for it. Essentially, duplicates won’t be allowed for standard pipeline.The issue occurs because
ConfigureHttpClientDefaults(x => x.AddStandardResilienceHandler())registers a single resilience pipeline with a namestandardthat is then shared by bothSucceedingClientandFailingClient. As a result, they both share the circuit breaker of thestandardpipeline.When we use two separate statements to register the resilience pipeline
we create two distinct resilience pipelines named
SucceedingClient-standardandFailingClient-standard, and each of them has its own circuit breaker. Each of these pipelines is then used by the respectiveHttpClients.The reason for that is how
AddStandardResilienceHandler()andConfigureHttpClientDefaults()methods work.AddStandardResilienceHandler()relies onIHttpClientBuilder.Namewhen registering and resolving the corresponding resilience pipeline. Whenever you callAddStandardResilienceHandler()you register a resilience pipeline for the givenIHttpClientBuilder.Name, and when you create anHttpClientthe corresponding resilience pipeline is retrieved by the name or type of theHttpClient, i.e. by the givenIHttpClientBuilder.Name.ConfigureHttpClientDefaults()executes its lambdax => x.AddStandardResilienceHandler()only once, and thexargument is anIHttpClienBuilderwith theName=null, i.e. it doesn’t have information about the actual name ofHttpClient. It doesn’t execute the lambda for each registeredHttpClient. As a result, we have a single resilience pipeline (and hence a single shared circuit breaker) which is applied to allHttpClients regardless of their name or type.@JamesNK Can
ConfigureHttpClientDefaults(Action<IHttpClientBuilder> configure)be updated to provide the actual name/type of theHttpClientbeing resolved? As I understand, currently itsconfigureaction is executed immediately whenConfigureHttpClientDefaults()is invoked, therefore it doesn’t have the actualNameofHttpClient. And to be able to provide theNameit should be updated to execute theconfigureaction when the actualHttpClientis being resolved/created, i.e. when there is information about its type/name. Can we updateConfigureHttpClientDefaults()in a such way?Workaround
@sveinungf As a workaround, you can configure the standard resilience pipeline to be applied per HTTP request authority:
If you do so, you’ll have distinct resilience pipelines per each HTTP request authority. It’ll solve the shared circuit breaker issue. You can even specify a custom criteria how to select resilience pipelines using the method
SelectPipelineBy.