traefik: Service status successful even if one or all servers/services failed

Do you want to request a feature or report a bug?

Bug

What did you do?

I created a service with a weighted service (Weighted Round Robin) and health check.

What did you expect to see?

In the GUI I expected to see the status failed or warning when at least one service/server status failed. For example (see config below): If server s1.test.com failed, I expexted to see the status of “http.services.app-2” equal to warning (since there is one more service available) and the status of “http.services.app-2v1” equal to failed (since all servers are failed).

What did you see instead?

The status of both services is successful.

Output of traefik version: (What version of Traefik are you using?)

2.1.1

What is your environment & configuration (arguments, toml, provider, platform, …)?

## Static configuration
[entryPoints]
    [entryPoints.web]
        address = ":80"
    [entryPoints.web-secure]
        address = ":443"
    [entryPoints.ping]
        address = ":8082"

[ping]
    entryPoint = "ping"

[providers]
    [providers.file]
        directory = "/etc/traefik/config-enabled"
        watch = true

[api]
    dashboard = true


[serversTransport]
    insecureSkipVerify = true
    [serversTransport.forwardingTimeouts]
        dialTimeout = "5s"
        responseHeaderTimeout = "1s"

[log]
    filePath = "/var/log/traefik.log"
    level = "WARN"

[accessLog]
    filePath = "/var/log/traefik-access.log"

[certificatesResolvers.le.acme]
    email = "support@mydomain.tld"
    storage = "/etc/traefik/acme/acme.json"
    [certificatesResolvers.le.acme.httpChallenge]
        entryPoint = "web"
[http.routers.domain-2]
    entryPoints = ["web"]
    rule = "Host(`domain-2.test.com`)"
    service = "app-2"

[http.services.app-2]
    [[http.services.app-2.weighted.services]]
        name = "app-2v1"
        weight = 3
    [[http.services.app-2.weighted.services]]
        name = "app-2v2"
        weight = 1

  [http.services.app-2v1]
      [http.services.app-2v1.loadBalancer]
          [http.services.app-2v1.loadBalancer.healthCheck]
              path = "/check"
              hostname = "domain-2.test.com"
              interval = "10s"
              timeout = "5s"
          [[http.services.app-2v1.loadBalancer.servers]]
              url = "http://s1.test.com/"

  [http.services.app-2v2]
      [http.services.app-2v2.loadBalancer]
          [http.services.app-2v2.loadBalancer.healthCheck]
              path = "/check"
              hostname = "domain-2.test.com"
              interval = "10s"
              timeout = "5s"
          [[http.services.app-2v2.loadBalancer.servers]]
              url = "http://s2.test.com/"

If applicable, please paste the log output in DEBUG level (--log.level=DEBUG switch)

time="2019-12-31T17:09:08+01:00" level=warning msg="Health check failed, removing from server list. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Weight: 1 Reason: receive$
time="2019-12-31T17:09:08+01:00" level=error msg="server not found"
time="2019-12-31T17:09:17+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:18+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:18+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"
time="2019-12-31T17:09:27+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:28+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:28+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"
time="2019-12-31T17:09:37+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:38+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:38+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"
time="2019-12-31T17:09:47+01:00" level=debug msg="Refreshing health check for backend: app-2v2@file"
time="2019-12-31T17:09:48+01:00" level=debug msg="Refreshing health check for backend: app-2v1@file"
time="2019-12-31T17:09:48+01:00" level=warning msg="Health check still failing. Backend: \"app-2v1@file\" URL: \"http://s1.test.com/\" Reason: received error status code: 504"

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 13
  • Comments: 37 (10 by maintainers)

Most upvoted comments

Thanks for the bug report, we would love community support to address it.

To summarize, the status in the Servers section represents the health check result of the servers, while the status of the Service Details section represents the configuration status (is there some issue reading the configuration). Those are two different statuses. Our proposal is to rename the Status of the Service Details section to Configuration in the UI. This should also be changed to the router & middleware panels.

298101082-41eeb68a-1844-4c90-a654-00fa766b8fc6 (1)

If you or another community member would like to build it, let us know, and we will work with you to make sure you have all the information needed so that it can be merged.

We prefer to work with our community members at the beginning of the design process so that we can make sure that we are aligned and can move quickly with the review and merge process. Let us know here or create a PR before you start, and we will work with you there.

Don’t forget to check out the contributor docs and to link the PR to this issue. If not, we will leave the issue open to any interested community member.

I think replacing the status section with configuration as keyword would be a nice step towards some more clarity on the topic. That would also be valid for routers and middlewares and everywhere where the “Success” status is used while referring to configuration matters.

At the same time, I think there is still a need for a “global” view, to examine everything at first glance. Some sort of “service degraded” warning in the dashboard. Otherwise I think all the effort that did go into the pie graphs would be wasted. My idea (for now, pretty rough) would be to have a part of the graph as yellow for degraded services, while also mentioning them somehow in the UI.

EDIT: looking at the dashboard now, there is already the warning section in the UI, so there is a possibility that one could be used as well. I just don’t know if that would cause some unintended consequences as far as user understanding between configuration and service status. In addition, instead of success, I think the proper term for configuration would be valid (or a similar one). (We would also show two different indicators in the service page so it would be pretty clean anyway)

At the same time, I think there is still a need for a “global” view, to examine everything at first glance. Some sort of “service degraded” warning in the dashboard.

+1. It would be nice if the services with no healthy backends would at least indicate “warning” status on the dashboard. Right now, at a glance, everything appears green even when every backend of every service is down. Very misleading.

Is there any date to adress this issue?

Hello all, Here is a proposal to improve the situation:

  • First iteration

    1. In Service Details
    • rename STATUS to CONFIGURATION
    • rename success to valid / invalid
    1. In Servers
    • If a healthcheck is defined, display the status column, which should be renamed to healthcheck
    • If no healthcheck is defined, don’t display the status (healthcheck) column
  • Second iteration

    • Add a HEALTH status in Service Details, that would gather the health statuses from underlying services

WDYT?

Hello, I was wondering if there is any chance that this could get some love in the upcoming update(s)? @Romloader made a brilliant post about ways to improve this and bring more clarity into when a server is failing behind a service.

That’s not what I said 😃 Today, we have one status in services, which reflects configuration issues. My proposal is to add another status in services, for health issues, which would gather health status from underlying servers. But this can come in a second iteration, the first iteration would be #6117 (comment)

Ah ok, sorry I misunderstood. Yes, two sets of information (config and health) would be great, thanks! I would kindly suggest making the health-related one more prominent (above the fold at least) because it will be dynamic and the first thing that people would check when assessing whether the service works.

Thanks!

I have the same problem. The service has only one server in the backend, but the service is displayed as up even when the server is down. I found this:

To propagate status changes (e.g. all servers of this service are down) upwards, HealthCheck must also be enabled on the parent(s) of this service.

How can I do this…? Currently I have a docker container with these labels: `labels:

  • traefik.enable=true
  • traefik.http.routers.vaultwarden.rule=Host(********)
  • traefik.http.routers.vaultwarden.entryPoints=https
  • traefik.http.services.app-vaultwarden.loadbalancer.healthcheck.path=/`

Same problem with v2.5.7 docker image. Any update/progress on this?

Hey, its missleading display in the UI. I have to admit that.

The Service Status is decoupled from the Server Status basically. We are thinking of how we can make that easier and better understandable