resilience4j: CircuitBreaker: How to treat exceptions as a success

Hi,

I have the following setup:

  1. A 3rd party service (runs on localhost:8090/sample) and it always returns bad request (400).
@RestController
@RequestMapping
public class SampleController {
    @GetMapping("/sample")
    public ResponseEntity helloWorld(HttpServletResponse response) {
        System.out.println("Request received");
        return ResponseEntity.badRequest().body("{\"message\":\"this is a bad request\"}");
    }
}
  1. In my main application where I am using circuit breaker, I am using RestTemplate to communicate it with the 3rd party service. I have ignored org.springframework.web.client.HttpClientErrorException from the circuit breaker.
  2. The circuit breaker annotation is kept at method level like below:
    @CircuitBreaker(name = "callOtherService", fallbackMethod = "errorExternal")
    public String callOtherService() {
        System.out.println("Call received");
        return restTemplate.getForObject(URI.create("http://localhost:8090/sample"), String.class);
    }

public String errorExternal(Throwable throwable) {
        return "This is error from fallback triggered by external error";
    }
  1. I am using following configuration (see callOtherService section)
resilience4j.circuitbreaker:
  configs:
    default:
      registerHealthIndicator: true
      ringBufferSizeInClosedState: 4
      ringBufferSizeInHalfOpenState: 2
      automaticTransitionFromOpenToHalfOpenEnabled: true
      waitDurationInOpenState: 20s
      failureRateThreshold: 50
      eventConsumerBufferSize: 10
      ignoreExceptions:
       - com.resilience4j.exception.BusinessException
       - feign.FeignException
      recordExceptions:
        - java.net.SocketTimeoutException
        - java.net.ConnectException
    shared:
      registerHealthIndicator: true
      ringBufferSizeInClosedState: 4
      ringBufferSizeInHalfOpenState: 2
      waitDurationInOpenState: 20s
      failureRateThreshold: 50
      eventConsumerBufferSize: 10
      ignoreExceptions:
        - com.resilience4j.exception.BusinessException
  instances:
    callOtherService:
      baseConfig: default
      ringBufferSizeInClosedState: 4
      registerHealthIndicator: true
      ignoreExceptions:
        # We need to ignore 4xx errors returned by the server as they are valid business case.
        - org.springframework.web.client.HttpClientErrorException
      recordExceptions:
        # We need to record all http errors
        - org.springframework.web.client.RestClientException

Steps to reproduce:

  1. Call the exposed endpoint 5-6 times. Verify the 3rd party service gets the call every time.
  2. Shut down the 3rd party service.
  3. Call the exposed endpoint again for 4 times. Check /actuator/health endpoint and you’ll see that the circuit is OPEN now.
  4. Wait for 20 sec.
  5. Now health endpoint says circuit is HALF_OPEN.
  6. Call exposed endpoint again for 2 times.
  7. Check health endpoint and verify circuit is OPEN again.
  8. Turn ON the 3rd party service.
  9. Verify from health endpoint if circuit is HALF_OPEN.
  10. Call the exposed endpoint. The call gets successful to 3rd party 2 times (size of ring buffer in half open state).
  11. Health endpoint still HALF_OPEN.
  12. All the calls to exposed endpoint are short-circuited and no call is attempted to the 3rd party.
  13. The circuit breaker is never CLOSED again. And there is no way to recover from this situation apart from restart the server using the circuit breaker.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 31 (24 by maintainers)

Commits related to this issue

Most upvoted comments

But I like the idea to add something like

recordExceptionsAsSuccess:
    - org.springframework.web.client.HttpClientErrorException

The fallback mechanism doesn’t take the CircuitBreakerConfig into account. It catches the exceptions which are part of the signature of your fallback method. Currently it’s like a try/catch. If your fallback uses Throwable/Exception it catches everything. You could narrow down the scope to more specific exceptions in your fallback method so that it does not catch ResourceNotFoundExceptions anymore. Or rethrow a ResourceNotFoundException in your fallback method. You can have multiple fallback methods with the same name, but with different signatures (exceptions). Comparable to different catch blocks.

But it is not necessary for an endpoint to return 200 to show that it is running properly. If any endpoint is returning 400 (valid case if the request fails validation error) that means also that the endpoint is working, right? As per me all the 4xx errors should be treated as successful call and only 5xx should be considered as failure because the 5xx ones are the ones which are related to server health.

It really depends on the signature of your fallback methods. If you catch Exception, the fallback method is called on every execption, not only when the CircuitBreaker is OPEN.

Usually it only makes sense to catch only CallNotPermittedException in your fallback-method of your CircuitBreaker. But we wanted to be as flexible as possible.

Thank a lot for your help! That way worked perfectly, the problem was solved