traefik: Gateway errors during rolling deployment w/sticky sessions

I do not consider this issue resolved. I’m glad there’s a workaround, but the underlying issue should be fixed. Or at the very least, the workaround should be documented.

Here’s a simple setup to reproduce the errors.

docker-stack.yml file:

version: "3.7"
services:
    
    app:
        image: traefik/whoami
        
        networks:
            - traefik_network
        
        ports:
            - 80
      
        deploy:
            replicas: 1
            update_config:
                order: start-first
                parallelism: 1

            labels:
                - "traefik.http.routers.app.rule=PathPrefix(`/`)"
                - "traefik.http.services.app.loadbalancer.server.port=80"
                - "traefik.http.services.app.loadBalancer.sticky.cookie=true"

                - "traefik.http.middlewares.retry-mw.retry.attempts=3"
                - "traefik.http.routers.app.middlewares=retry-mw@docker"
    
    
    load-balancer:
        image: traefik:v2.4.2
        
        command:            
            - --providers.docker=true
            - --providers.docker.swarmMode=true
            - --providers.docker.network=traefik_network
            - --entryPoints.app.address=:80
            
            - --log.level=DEBUG
            
        networks:
            - traefik_network
            
        ports:
            - "80:80"
        
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock


networks:
    traefik_network:
        external: true

Setup commands:

docker swarm init
docker network create -d overlay traefik_network --attachable
docker stack deploy -c docker-stack.yml demo

Testing commands: docker service update --force demo_app IMMEDIATELY after previous command, run: ab -n 5000 -s 120 localhost/

Adjust the 5000 parameter based on how fast your computer handles the incoming requests. If you run those two commands close enough together, the requests will initially go to the first container, but then at some point it’ll switch over to the new one.

During the switchover, there will be errors where Traefik tries to send requests to the container that just shut down rather than using the new container that should already be working correctly. You can also see from the logs that the Retry middleware just keeps trying the ip address of the old container rather than switching over to the new one.

Typically when I’ve been testing it, I see somewhere from 10-30 failed requests each time. If you inspect the docker logs for the traefik container, you can find the errors easily by searching for “gateway”. They’re typically a mixture of 502 and 504 errors. You can also find the retries easily if you search for “New attempt”.

Let me know if you have any other questions or any difficulties with reproducing the issue.

@Alvise88 Can you elaborate on that at all? Why is it pure luck? And what side effects?

My naive assumption is that this change allows Traefik to more quickly see the updates from Docker, so it stops using the old container and starts using the new one fast enough that there are no errors.

That’s not how the delay feature works. If your swarm is running multiple replicas, the delay is how much time the swarm will wait in between starting up each new replica.

In my reproduction sample code above, there is only one replica, so adding a delay option doesn’t do anything at all. The way 100% uptime is supposed to be preserved is because of the start-first order. This ensures that the new container is up and running before the swarm takes the old one offline.

In my real code, I also have a docker healthcheck configured, so I know for sure that the new container is successfully accepting requests before the old ones are removed.

@allanjackson Similar situation here, I also see “Bad Gateway” often but not only when updating, it occurs independent of service updates and independent of task crashes. This happens with the latest 2.3 here.

I also use retry middleware and stickiness basically the same way like you (without the traefik lb healthCheck).

It happened around 260 times in 48 hours (50 req/s average)