traefik: Traefik healthcheck should not pass by default for newly discovered backends
Do you want to request a feature or report a bug?
Bug
What did you do?
Background: we deploy changes to our marathon masters and mesos agents by replacing the AWS instances with CloudFormation rolling updates. We co-locate Traefik on each of the mesos agents. Production environment is about 200 marathon apps, 800 containers over 25 EC2 instances, serving generally around 1K to 4K req/s.
During a rolling update to production, we drop almost 1% of requests. Our aim is to not drop any requests (but we might settle for 0.01%) so in a test environment, we’re working through various points of lifecycle of marathon tasks and Traefik configuration to analyse the reasons for dropped requests.
With the help of the great manual - https://docs.traefik.io/user-guide/marathon/ - we have made some progress:
- we have enabled readiness checks
- we have patched the mesos containerizer source to implement an extra shutdown delay between entering TASK_KILLING and actually running “docker stop” (as an alternative to requiring to modify the signal handling in 200 microservices)
- this has given perfect results of zero dropped requests when scaling or deploying a marathon app.
However, when doing a rolling update of the mesos agents the task lifecycle is different.
Tasks are evacuated from each instance and recreated by marathon on other agents without a deployment occurring … which means that there are no readiness checks to prevent new tasks going live in Traefik before they are ready. This means we receive a lot of 502 errors when Traefik tries to send requests to the newly added backends.
To try and deal with this cause of dropped requests and 502 errors we have enabled Traefik healthchecks on our test service. This gives a significant improvement but we’re still dropping some requests.
It seems that this is because when Traefik is triggered (by a marathon event) to reconfigure itself and discovers a new backend, it initially puts it in active rotation … in other words the initial state of the Traefik healthcheck defaults to “passing”.
I think this is arguably a bug and that any newly discovered backend with a healthcheck should initially be presumed to be failing its healthcheck until such time as the healthcheck actually passes.
Output of traefik version: (What version of Traefik are you using?)
Version: v1.7.2
Codename: maroilles
Go version: go1.11.1
Built: 2018-10-04_01:44:36PM
OS/Arch: linux/amd64
What is your environment & configuration (arguments, toml, provider, platform, …)?
Some uninteresting stuff like log headers removed
logLevel = "INFO"
defaultEntryPoints = ["http"]
[entryPoints]
[entryPoints.http]
address = ":80"
[traefikLog]
format = "json"
[accessLog]
format = "json"
# API definition
[api]
entryPoint = "traefik"
dashboard = true
debug = false
[api.statistics]
recentErrors = 10
# Metrics definition
[metrics]
[metrics.datadog]
address = "xxx.yyy.50.75:8125"
pushInterval = "10s"
[ping]
entryPoint = "traefik"
[marathon]
endpoint = "http://mesos.our.domain:8080"
watch = true
domain = "our.domain"
exposedByDefault = false
respectReadinessChecks = true
If applicable, please paste the log output in DEBUG level (--logLevel=DEBUG switch)
Test scenario
- one test marathon application
mock-webappwith 3 marathon tasks on 3 out of 5 different mesos agents - mock-webapp has an artificial start-up delay of 15s before it opens the socket to listen to simulate java app startup
- mock-webapp then answers requests with an artificial delay of 100-200ms to simulate real world service latency
- AWS ALB distributing HTTP requests over 5 Traefik instances, one on each mesos agent
- we are firing load at the ALB -> Traefik at 75 req/s
- we enabled debug logging on one Traefik process and here is an extract excluding the successful requests (the whole thing is a few thousand lines)
- traefik healthchecks are enabled every 10s
<marathon event received>
12:32:53.319 Creating server server-mock-webapp-mock-webapp-1809d998-3b4d-11e9-a7e1-2ef24d6dcd47 at http://xxx.yyy.52.15:31658 with weight 1
12:32:53.319 Creating server server-mock-webapp-mock-webapp-f726fe2d-3b54-11e9-bc7d-4a6ed5e7b0d0 at http://xxx.yyy.50.75:31518 with weight 1
12:32:53.319 Creating server server-mock-webapp-mock-webapp-65eecc6b-3b4d-11e9-a7e1-2ef24d6dcd47 at http://xxx.yyy.52.119:31928 with weight 1
12:32:53.319 Creating backend backend-mock-webapp
12:32:53.319 Setting up backend health check [Hostname: Headers: map[] Path: /?max_latency=0.1 Port: 0 Interval: 10s]
12:32:53.319 Wiring frontend frontend-mock-webapp to entryPoint http
12:32:53.320 Creating route route-host-mock-webapp Host: mock-webapp.domain
12:32:53.326 Stopping current health check goroutines of backend: backend-mock-webapp
12:32:53.326 Initial health check for backend: "backend-mock-webapp"
12:32:53.586 '502 Bad Gateway' caused by: dial tcp xxx.yyy.50.75:31518: connect: connection refused
<there are also a few more 502 errors>
12:32:54.000 Health check failed: Remove from server list. Backend: "backend-mock-webapp" URL: "http://xxx.yyy.50.75:31518" Reason: HTTP request failed: Get http://xxx.yyy.50.75:31518/%3Fmax_latency=0.1: dial tcp xxx.yyy.50.75:31518: connect: connection refused
12:33:04.000 Refreshing health check for backend: backend-mock-webapp
12:33:04.001 Health check still failing. Backend: "backend-mock-webapp" URL: "http://xxx.yyy.50.75:31518" Reason: HTTP request failed: Get http://xxx.yyy.50.75:31518/%3Fmax_latency=0.1: dial tcp xxx.yyy.50.75:31518: connect: connection refused
12:33:09.007 Received provider event type: health_status_changed_event, event: &{health_status_changed_event 2019-02-28T12:33:09.005Z /paas/mock-webapp 2019-02-28T11:36:31.271Z %!s(bool=true)}
<traefik reconfigures itself again 2 times, no changes>
12:33:14.000 Refreshing health check for backend: backend-mock-webapp
12:33:14.571 Health check up: Returning to server list. Backend: "backend-mock-webapp" URL: "http://xxx.yyy.50.75:31518"
So you can see that for almost 700ms between 12:32:53.319 and 12:32:54.000 the backend xxx.yyy.50.75:31518 is presumed to be healthy and is being sent requests, until such time as the first healthcheck request fails, and then it is marked unhealthy.
During this period, we unavoidably get 502 errors because Traefik is sending requests to a marathon task which is not yet listening on the port.
At 12:33:09 you can see the task go healthy in marathon. At 12:33:14 this traefik node sees the service as healthy and puts it back in rotation
Here’s a log of all the 502 errors received across all 5 Traefik + mesos-agent nodes:
StartUTC OriginStatus Host Duration
12:32:53.316 502 mesos-agent-private-4 1074858
12:32:53.364 502 mesos-agent-private-2 957254
12:32:53.396 502 mesos-agent-private-3 1081129
12:32:53.421 502 mesos-agent-private-1 588835
12:32:53.476 502 mesos-agent-private-3 1041597
12:32:53.499 502 mesos-agent-private-1 753561
12:32:53.583 502 mesos-agent-private-6 2501728
12:32:53.621 502 mesos-agent-private-2 1017768
12:32:53.633 502 mesos-agent-private-4 1569354
12:32:53.653 502 mesos-agent-private-4 669198
12:32:53.729 502 mesos-agent-private-2 935705
12:32:53.754 502 mesos-agent-private-6 2740235
12:32:53.757 502 mesos-agent-private-3 1081155
12:32:53.869 502 mesos-agent-private-3 1064540
12:32:53.933 502 mesos-agent-private-6 1709245
12:32:53.968 502 mesos-agent-private-4 712834
12:32:54.032 502 mesos-agent-private-2 1064561
12:32:54.033 502 mesos-agent-private-4 772085
12:32:54.073 502 mesos-agent-private-2 984483
12:32:54.140 502 mesos-agent-private-3 1175981
12:32:54.376 502 mesos-agent-private-3 1141847
12:32:54.461 502 mesos-agent-private-3 1017805
You can see that the total period of errors is around 1145ms - presumably each individual node received the marathon event at a slightly different times and took a different amount of time to reconfigure and schedule the first healthcheck.
Before and after this period, we receive only 200 responses.
what could be done differently?
I think that the ideal behaviour here would be that the new backend that has not yet been healthchecked is marked as unhealthy until the first successful healthcheck result is processed.
- would you agree?
- do you think this is feasible to fix?
- is there any other way that we could work around this issue?
Thanks and sorry for the very long bug report!
About this issue
- Original URL
- State: open
- Created 8 years ago
- Reactions: 14
- Comments: 21 (6 by maintainers)
@andyspiers based on our and additional discussions I’ve had with other maintainers, it seems reasonable to allow for the default health check state to be unhealthy as you requested. Popular management platforms (like Kubernetes and Docker) also default to that state.
Additionally, there doesn’t seem to be any harm in changing the default behavior (as opposed to making it configurable) except for a slight increase in delay. If that’s really problematic to anybody, we can contemplate adding parameters to tune health check behavior (e.g., periodicity and initial delay) where we do not have them yet. This would be beneficial beyond the issue at hand.
I can drive the necessary change myself when I have some free bandwidth. If anyone is happy to pick up the PR effort, however, I won’t resist much. 😃
Thanks!
I am testing a blue/green deployment scenario using the docker backend, aiming for zero dropped requests, and am hitting this same issue. The new docker container is immediately considered healthy by traefik and is sent requests before the container is fully initialized and able to process requests.
The proposed fix (allowing for the default health check state to be unhealthy) would solve this problem for me as well.
Is there appetite to change the default behavior here from up to down? Looking through the load balancer code I think it may be derived from this line:
https://github.com/traefik/traefik/blob/063f8fae790c4aad4f44e54645d3972679aca6a2/pkg/healthcheck/healthcheck.go#L111
Alternatively, this could be implemented closer to the Docker provider
👋 Curious if anyone knows any workarounds to resolve brief moments of requests being routed to unhealthy servers before the health check runs?
AFAIK @TeNNoX nobody has started work on this
cool - thanks so much @timoreimann for digging into this 😃