traefik: Service status successful even if one or all servers/services failed
Do you want to request a feature or report a bug?
Bug
What did you do?
I created a service with a weighted service (Weighted Round Robin) and health check.
What did you expect to see?
In the GUI I expected to see the status failed or warning when at least one service/server status failed. For example (see config below): If server s1.test.com failed, I expexted to see the status of “http.services.app-2” equal to warning (since there is one more service available) and the status of “http.services.app-2v1” equal to failed (since all servers are failed).
What did you see instead?
The status of both services is successful.
Output of traefik version: (What version of Traefik are you using?)
2.1.1
What is your environment & configuration (arguments, toml, provider, platform, …)?
## Static configuration
[entryPoints]
[entryPoints.web]
address = ":80"
[entryPoints.web-secure]
address = ":443"
[entryPoints.ping]
address = ":8082"
[ping]
entryPoint = "ping"
[providers]
[providers.file]
directory = "/etc/traefik/config-enabled"
watch = true
[api]
dashboard = true
[serversTransport]
insecureSkipVerify = true
[serversTransport.forwardingTimeouts]
dialTimeout = "5s"
responseHeaderTimeout = "1s"
[log]
filePath = "/var/log/traefik.log"
level = "WARN"
[accessLog]
filePath = "/var/log/traefik-access.log"
[certificatesResolvers.le.acme]
email = "support@mydomain.tld"
storage = "/etc/traefik/acme/acme.json"
[certificatesResolvers.le.acme.httpChallenge]
entryPoint = "web"
[http.routers.domain-2]
entryPoints = ["web"]
rule = "Host(`domain-2.test.com`)"
service = "app-2"
[http.services.app-2]
[[http.services.app-2.weighted.services]]
name = "app-2v1"
weight = 3
[[http.services.app-2.weighted.services]]
name = "app-2v2"
weight = 1
[http.services.app-2v1]
[http.services.app-2v1.loadBalancer]
[http.services.app-2v1.loadBalancer.healthCheck]
path = "/check"
hostname = "domain-2.test.com"
interval = "10s"
timeout = "5s"
[[http.services.app-2v1.loadBalancer.servers]]
url = "http://s1.test.com/"
[http.services.app-2v2]
[http.services.app-2v2.loadBalancer]
[http.services.app-2v2.loadBalancer.healthCheck]
path = "/check"
hostname = "domain-2.test.com"
interval = "10s"
timeout = "5s"
[[http.services.app-2v2.loadBalancer.servers]]
url = "http://s2.test.com/"
If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)
time="2019-12-31T17:09:08+01:00" level=warning msg="Health check failed, removing from server list. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Weight: 1 Reason: receive$
time="2019-12-31T17:09:08+01:00" level=error msg="server not found"
time="2019-12-31T17:09:17+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:18+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:18+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"
time="2019-12-31T17:09:27+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:28+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:28+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"
time="2019-12-31T17:09:37+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:38+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:38+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"
time="2019-12-31T17:09:47+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:48+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:48+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"
About this issue
- Original URL
- State: open
- Created 7 years ago
- Reactions: 13
- Comments: 37 (10 by maintainers)
I think replacing the
statussection withconfigurationas keyword would be a nice step towards some more clarity on the topic. That would also be valid for routers and middlewares and everywhere where the “Success” status is used while referring to configuration matters.At the same time, I think there is still a need for a “global” view, to examine everything at first glance. Some sort of “service degraded” warning in the dashboard. Otherwise I think all the effort that did go into the pie graphs would be wasted. My idea (for now, pretty rough) would be to have a part of the graph as yellow for degraded services, while also mentioning them somehow in the UI.
EDIT: looking at the dashboard now, there is already the warning section in the UI, so there is a possibility that one could be used as well. I just don’t know if that would cause some unintended consequences as far as user understanding between configuration and service status. In addition, instead of
success, I think the proper term for configuration would bevalid(or a similar one). (We would also show two different indicators in the service page so it would be pretty clean anyway)+1. It would be nice if the services with no healthy backends would at least indicate “warning” status on the dashboard. Right now, at a glance, everything appears green even when every backend of every service is down. Very misleading.
Is there any date to adress this issue?
Hello all, Here is a proposal to improve the situation:
First iteration
STATUStoCONFIGURATIONsuccesstovalid/invalidstatuscolumn, which should be renamed tohealthcheckstatus(healthcheck) columnSecond iteration
HEALTHstatus in Service Details, that would gather the health statuses from underlying servicesWDYT?
Hello, I was wondering if there is any chance that this could get some love in the upcoming update(s)? @Romloader made a brilliant post about ways to improve this and bring more clarity into when a server is failing behind a service.
Great 😃 I completed the proposal accordingly https://github.com/traefik/traefik/issues/6117#issuecomment-2236476215
Ah ok, sorry I misunderstood. Yes, two sets of information (config and health) would be great, thanks! I would kindly suggest making the health-related one more prominent (above the fold at least) because it will be dynamic and the first thing that people would check when assessing whether the service works.
Thanks!
I have the same problem. The service has only one server in the backend, but the service is displayed as up even when the server is down. I found this:
How can I do this…? Currently I have a docker container with these labels: `labels:
********)Same problem with v2.5.7 docker image. Any update/progress on this?
Hey, its missleading display in the UI. I have to admit that.
The Service Status is decoupled from the Server Status basically. We are thinking of how we can make that easier and better understandable