traefik: websocket errors after upgrading traefik from 3.2.3 to 3.2.4 or 3.3.0

Welcome!

  • Yes, I’ve searched similar issues on GitHub and didn’t find any.
  • Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.

What did you do?

I am running MinIO behind traefik using docker compose, and after upgrading today to 3.2.4 or 3.3.0 MinIO fails with websocket errors when trying to list/browse objects in a bucket through the console web interface. Downgrading to 3.2.3 fixes it again with the following compose file listed below. What broke!?!

What did you see instead?

websocket connection error in browser developer console and nothing is listed…

What version of Traefik are you using?

3.2.4

What is your environment & configuration?


services:
  minio1:
    image: minio/minio:RELEASE.2024-12-18T13-15-44Z
    restart: unless-stopped
    volumes:
      - /opt/minio:/opt/minio
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ADMIN_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ADMIN_PASS}
      MINIO_VOLUMES: "/opt/minio"
      MINIO_PROMETHEUS_AUTH_TYPE: public
    command:
      - server
      - --address
      - ":9000"
      - --console-address
      - ":9001"
    healthcheck:
      test: [ "CMD", "mc", "ready", "local" ]
      interval: 5s
      timeout: 5s
      retries: 5
    labels:
      - "traefik.enable=true"
      ## minio-console
      - "traefik.http.routers.minio-console.rule=Host(`minio-console-osl.example.com`)"
      - "traefik.http.routers.minio-console.entrypoints=web,websecure"
      - "traefik.http.routers.minio-console.tls=true"
      - "traefik.http.routers.minio-console.service=minio-console"
      - "traefik.http.services.minio-console.loadbalancer.server.port=9001"
      ## minio-api
      - "traefik.http.routers.minio-api.rule=HostRegexp(`[host:.+]`)"
      - "traefik.http.routers.minio-api.entrypoints=web,websecure"
      - "traefik.http.routers.minio-api.tls=true"
      - "traefik.http.routers.minio-api.service=minio-api"
      - "traefik.http.services.minio-api.loadbalancer.server.port=9000"

  traefik:
    # image: "traefik:v3.2.3"
    image: "traefik:v3.2.4"
    container_name: "traefik"
    restart: unless-stopped
    command:
      - "--log.level=DEBUG"
      - "--api"
      - "--api.insecure=true"
      - "--api.dashboard=true"
      - "--accesslog=true"
      - "--accesslog.format=json"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.traefik.address=:8080"
      - "--providers.file.directory=/traefik-conf/"
      - "--providers.file.watch=true"
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.addEntryPointsLabels=true"
      - "--metrics.prometheus.addServicesLabels=true"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - $PWD/traefik-conf/dynamic-conf.toml:/traefik-conf/dynamic-conf.toml
      - $PWD/ssl:/ssl
      - /var/log/:/var/log/
    depends_on:
      - minio1

If applicable, please paste the log output in DEBUG level

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 41
  • Comments: 30 (8 by maintainers)

Most upvoted comments

Hey there,

We have found the root cause of the issue. We’re in the process of fixing it and releasing new versions.

In the meantime, you can use the latest version of Traefik if you set the following environment variable: GODEBUG=http2xconnect=0.

We’ll update the issue with the PR that fixes it.

Same goes for HomeAssistant, could not access it through my Traefik URL but local IP worked fine. After downgrading to 3.2.3 it works fine again.

http:
  routers:
    home-assistant:
      entryPoints:
        - https
      service: home-assistant-service
      rule: "Host(`home.domain.nl`)"

  services:
    home-assistant-service:
      loadBalancer:
        servers:
          - url: http://192.168.1.9:8123
        passHostHeader: true

+1 same issue w/ home-assistant

Not trying to be too impatient here, but I see no updated docker images, neither on Docker Hub or GHCR. The GODEBUG=http2xconnect=0 workaround works, but it’d be nice to have an updated image that doesn’t need the workaround, if possible 🙏

FYI v3.3.1 is now available in docker registry.

Not trying to be too impatient here, but I see no updated docker images, neither on Docker Hub or GHCR. The GODEBUG=http2xconnect=0 workaround works, but it’d be nice to have an updated image that doesn’t need the workaround, if possible 🙏

Hey there,

Thanks for raising this issue. We are analyzing it.

We’ll make sure to keep you updated following up on this ticket as soon as we have more information to share.

The Official Docker images are built and published by Docker itself (this is not a classic Docker Hub image).

The images were built an hour ago, but the deployment takes some time.

https://github.com/docker-library/meta/actions/runs/12661032071

You have to wait a bit.

The tag v3.3 has been deployed 6 minutes ago, latest and v3 8 minutes ago, I think the other will come soon.

Edit: all the tags are available now.

v3, v3.3 latest, etc. are rolling tags, you should explicitly pull (force) them to get the update.

All those tags are currently aliases of v3.3.1: https://github.com/docker-library/official-images/blob/adc7658c371bd8bf3f78d53d45195f21696c5e00/library/traefik#L24

In that case, updating those tags must be waiting in a queue somewhere, because those tags are currently definitely pointing at 3.3.0.

You can see it by going to https://hub.docker.com/_/traefik/tags, make a note of the digest for v3.3.0 linux/amd64 (8fb30ac...), then look at the digest voor v3 linux/amd64, which is identical. Also, the v3 tag was pushed hours before the v3.3.1 tag was.

None of this is your fault though. We’ll just have to wait a little longer.

EDIT: Oh, and none of the tags pushed 2 hours ago actually point at anything, so 3.3.1, 3.2.5 and 2.11.18 can’t currently be pulled at all.

FYI v3.3.1 is now available in docker registry.

Excellent, thanks! I don’t know if you made that happen, but the various v3 tags (v3, v3.3, 3.3 and 3) still point at the v3.3.0 image. I’d be grateful if those could also be updated.

FYI, this seems to also affect 2.11.17: rolled back to 2.11.16 in the mean time

EDIT: duplicate of above https://github.com/traefik/traefik/issues/11405#issuecomment-2574220527

Closed by #11408.

3.3.0-rc2 was working correctly so must have been a regression introduced afterwards

I’m having the same issue after an upgrade to 2.11.17, rolling back to 2.11.16 fixes Home Assistant for me.

Confirmed. 3.2.4 and 3.3.0 broke my homelab. Back to 3.2.3 waiting for the fix.