traefik: Increased memory usage and memory leak in Traefik v3

After upgrading to Traefik 3 (v3.0.3) I noticed it uses a lot more memory (70 MB vs 5 GB) than v2 did. And it keeps increasing, indicating a possible memory leak.

image

Switching traefik container back to 2.11 fixes the issue.

I have 20 routers, 17 services and 12 middlewares (all HTTP). Most from docker container labels.

What version of Traefik are you using?

3.0.3

What is your environment & configuration?

traefik.yml

metrics:
  prometheus:
    entryPoint: metrics
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    buckets:
      - 0.1
      - 0.3
      - 1.2
      - 5.0
global:
  checkNewVersion: true
  sendAnonymousUsage: false
log:
  filePath: /var/log/traefik/traefik.log
  format: json
accessLog:
  filePath: /var/log/traefik/access.log
api:
  dashboard: true
  insecure: false
certificatesResolvers:
  le:
    acme:
      email: ***
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 0
      storage: /acme/acme.json
entryPoints:
  metrics:
    address: :8899
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: :443
    http:
      tls:
        certResolver: le
    forwardedHeaders:
      trustedIPs:
        - ***
ping:
  entryPoint: websecure
providers:
  docker:
    exposedByDefault: false
    endpoint: "unix:///var/run/docker.sock"
  file:
    directory: /etc/traefik
    watch: true

experimental:
  plugins:
    real-ip:
      moduleName: github.com/Paxxs/traefik-get-real-ip
      version: v1.0.3

dynamic.yml

http:
  middlewares:
    test1:
      chain:
        middlewares:
          - compression
          - real-ip-plugin

    test2:
      retry:
        attempts: 1
        initialinterval: 100ms

    redirect-to-www:
      redirectregex:
        regex: ^https://(?:www\.)?(.*)
        replacement: https://www.$${1}

    auth:
      basicAuth:
        users:
          - "***"

    compression:
      compress:
        excludedContentTypes:
          - text/event-stream

    security:
      headers:
        accessControlAllowOriginList:
          - "*"

    real-ip-plugin:
      plugin:
        real-ip:
          Proxy:
            - proxyHeadername: ***
              proxyHeadervalue: ***
              realIP: ***
              OverwriteXFF: true
            - proxyHeadername: "*"
              realIP: ***

  serversTransports:
    insecure:
      insecureSkipVerify: true

docker-compose.yml

networks:
  traefik:
    name: traefik
    enable_ipv6: false
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 192.168.3.0/24
          gateway: 192.168.3.1

services:
  traefik:
    #image: traefik:v2.11
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    networks:
      - traefik
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./config/:/etc/traefik/
      - ${DOCKER_DATA}/traefik/acme:/acme
      - ${DOCKER_DATA}/traefik/logs:/var/log/traefik
    environment:
      BASE_DOMAIN: ${BASE_DOMAIN}
      CF_ZONE_API_TOKEN: ${CF_ZONE_API_TOKEN}
      CLOUDFLARE_DNS_API_TOKEN: ${CF_ZONE_API_TOKEN}
    labels:
      - "traefik.docker.network=traefik"
      - "traefik.enable=true"
      - traefik.http.routers.traefik.rule=Host(`traefik.${BASE_DOMAIN}`)
      - traefik.http.routers.traefik.entrypoints=websecure
      - traefik.http.routers.traefik.service=api@internal
      
      # Forward auth
      - traefik.http.middlewares.auth-traefik.forwardauth.address=http://authentik-outpost:9000/outpost.goauthentik.io/auth/traefik
      - traefik.http.middlewares.auth-traefik.forwardauth.trustForwardHeader=true
      - traefik.http.routers.traefik.middlewares=security@file, test1@file, auth-traefik@docker
      # Route logout pages
      - traefik.http.routers.traefik-auth.middlewares=security@file, test1@file
      - traefik.http.routers.traefik-auth.rule=Host(`traefik.${BASE_DOMAIN}`) && PathPrefix(`/outpost.goauthentik.io/`)
      - traefik.http.routers.traefik-auth.service=authentik-proxy@docker
      - traefik.http.routers.traefik-auth.entrypoints=websecure
      - traefik.http.routers.traefik-auth.tls.domains[0].main=${BASE_DOMAIN}
      - traefik.http.routers.traefik-auth.tls.domains[0].sans=*.${BASE_DOMAIN}

      - com.centurylinklabs.watchtower.enable=true
      - com.centurylinklabs.watchtower.scope=daily

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 11
  • Comments: 53 (14 by maintainers)

Most upvoted comments

Same here; I had the same issue after upgrade from 2.11 to 3.2, memory usage reached up to >5 GB, the nodes were acting crazy, how did they miss this issue with zstd !

I set this to fix it for now as I can’t rollback to 2.11 especially after the CRD api changes.

apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: compress namespace: ****** spec: compress: defaultEncoding: gzip encodings: - gzip

I understand the reasoning, but if if I was a new Traefik user and it would use 5GB of RAM, I would no longer be a Traefik user.

V3 at the state of the 3.0 release would absolutely scare me away from it.

After running some tests, I can confirm that the memory usage is linked to the compression algorithms.

Here are the results I observed:

Without compression: Memory usage is around 2-3 MB. With Gzip enabled: Memory usage goes up to 20-30 MB. With Zstandard (Zstd) enabled: Memory usage is between 60-80 MB. With Brotli enabled: Memory usage increases to 350-500 MB. The new compression methods introduced in Traefik 3.x seem to require more memory compared to Gzip or no compression. Starting from version 3.2, there is a new feature that allows excluding certain compressions, which can help reduce memory usage. https://doc.traefik.io/traefik/v3.2/middlewares/http/compress/#encodings

I suggest testing with the 3.2 RC version, as the final release should arrive soon.

@deyanp @Janhouse I also had a memory leak with the v3 version, when I removed the compress middleware the memory leak disappeared. Maybe you can try it.

Closed by #11641.

There is still a memory issue with the zstd compression, not sure this issue can really be marked as solved.

I think the default option (zstd) having this issue wouldn’t let this be “solved”. Probably should revert the default to gzip.

I’d prefer to use zstd where possible since it yields slightly better performance/compression.

While I agree manually becoming aware of this issue, and having the knowledge to switch back to gzip, does resolve the symptoms, I have to consider the rock solid stability of 2.11 – I can’t risk encountering further potential tripwires within 3.x

@mattbha With v3.2 we recommend to configure the encodings you need thanks to the encodings option.

I am no longer experiencing OOM issues.

After testing v3.2 with gzip only, I confirm the issue is solved.

I’m using traefik v3.1.2 and I’m experiencing the same issue with a high memory consumption. No issue in traefik v2.

We also resolved it by switching the compression to gzip. As we have millions of requests per minute, our Traefik lasted for 2-3 minutes before hitting the 300 MB memory limit.

Looks like you should include stress testing so you can catch things like these before switching something to default. And thanks for the solution.

We disabled zstd compression in Traefik v3 after we run into memory leak issues.

After upgrading from v2 to v3, memory usage was six times higher than before, so our pods were OOM killed. We gave them more memory and it was fully utilized. After switching back to gzip, everything was back to normal.

Under a simple 100 user locust stress test I notice my traefik RAM usage (in a docker-compose stack) shot up to about 1.4gb before I stopped it image: traefik:3.2

When I changed the docker-compose to image: traefik:2.11 I hit about 140mb max

default settings everywhere, compression middleware enabled.

3.2 was choosing to use zstd 2.11 is using gzip of course

I cannot upgrade to 3.X from 2.11 due to this issue. Definitely appeared like a memory leak would (3.2 memory kept growing and growing under stress test, likely due to zstd).

Combined with the above problem analysis and solution, finally through the constraint of “preferential use of Gzip compression algorithm”, the same result is obtained in v3.2.0 and v2.11.13. Thank you very much!

@Janhouse I understand your concern, but compression is not enabled by default. So, a new Traefik user won’t immediately use 5GB of RAM unless they enable the compression middleware. That said, this is just my personal opinion. I’ll share your feedback with the other maintainers during the next triage and get back to you as soon as possible.

@Janhouse I wasn’t able to reproduce the leak, but I did reproduce the “larger memory footprint.”

Does enabling the different compression algos by default make any sense with the increased memory usage?

Personally, if we change this now, it might feel like a breaking change. Some users may have higher memory limits and prefer to use more memory with all the compression enabled.

For me, the main issue was that we couldn’t configure it, but this will be possible in v3.2.

Thank you all for your input and for sharing your experiences regarding the memory leak. After investigating on my end, I haven’t been able to reproduce the memory leak issue with any compression method, including br and zstd. While it’s true that compression does use additional memory, in my tests, memory usage returns to normal levels once traffic decreases.

However, there is one notable difference I would like to highlight between Traefik v2 and v3 regarding compression. In v2, only gzip was supported, whereas in v3, we added support for Brotli (br) and Zstandard (zstd). These newer compression algorithms are more memory-intensive compared to gzip. From my tests, I’ve observed that gzip consumes around 200MB, whereas Brotli and Zstandard can go up to 900MB.

To address this, a PR has been merged that introduces the ability to select which compression algorithms you want to enable. This feature will be available in the next major release, which is expected to be released very soon.

In the meantime, to further investigate this potential memory leak, could someone provide a complete reproducible case? This would greatly help us narrow down the issue. Ideally, here’s what we would need:

  • Full Traefik configuration (including any middleware or specific settings related to compression).
  • Details about the backends you’re using (e.g., type of services, behavior under load).
  • Traffic generation tool you’re using to simulate load (e.g., curl, wrk, or other testing frameworks) and the exact parameters used during the tests.
  • If possible, any metrics or logs showing memory usage over time.

The more detailed the scenario, the quicker we can identify and resolve the issue.

Hello, it’s currently my server configuration https://github.com/toutvukantabu/Homelab-traefik-authentik-crowdsec-server thanks a lot for your investigations!

Thank you all for your input and for sharing your experiences regarding the memory leak. After investigating on my end, I haven’t been able to reproduce the memory leak issue with any compression method, including br and zstd. While it’s true that compression does use additional memory, in my tests, memory usage returns to normal levels once traffic decreases.

However, there is one notable difference I would like to highlight between Traefik v2 and v3 regarding compression. In v2, only gzip was supported, whereas in v3, we added support for Brotli (br) and Zstandard (zstd). These newer compression algorithms are more memory-intensive compared to gzip. From my tests, I’ve observed that gzip consumes around 200MB, whereas Brotli and Zstandard can go up to 900MB.

To address this, a PR has been merged that introduces the ability to select which compression algorithms you want to enable. This feature will be available in the next minor release, which is expected to be released very soon.

In the meantime, to further investigate this potential memory leak, could someone provide a complete reproducible case? This would greatly help us narrow down the issue. Ideally, here’s what we would need:

  • Full Traefik configuration (including any middleware or specific settings related to compression).
  • Details about the backends you’re using (e.g., type of services, behavior under load).
  • Traffic generation tool you’re using to simulate load (e.g., curl, wrk, or other testing frameworks) and the exact parameters used during the tests.
  • If possible, any metrics or logs showing memory usage over time.

The more detailed the scenario, the quicker we can identify and resolve the issue.

I’ve been experiencing the same issue after upgrading to v3. Memory usage would just keep increasing until an OOM event. I found that when i set watch: false on the file provider, however, the problem went away. I’m not sure why exactly, as I haven’t had a chance to investigate further, but perhaps try setting that to false and see if it helps?