traefik: Increased memory usage and memory leak in Traefik v3
After upgrading to Traefik 3 (v3.0.3) I noticed it uses a lot more memory (70 MB vs 5 GB) than v2 did. And it keeps increasing, indicating a possible memory leak.
Switching traefik container back to 2.11 fixes the issue.
I have 20 routers, 17 services and 12 middlewares (all HTTP). Most from docker container labels.
What version of Traefik are you using?
3.0.3
What is your environment & configuration?
traefik.yml
metrics:
prometheus:
entryPoint: metrics
addEntryPointsLabels: true
addRoutersLabels: true
addServicesLabels: true
buckets:
- 0.1
- 0.3
- 1.2
- 5.0
global:
checkNewVersion: true
sendAnonymousUsage: false
log:
filePath: /var/log/traefik/traefik.log
format: json
accessLog:
filePath: /var/log/traefik/access.log
api:
dashboard: true
insecure: false
certificatesResolvers:
le:
acme:
email: ***
dnsChallenge:
provider: cloudflare
delayBeforeCheck: 0
storage: /acme/acme.json
entryPoints:
metrics:
address: :8899
web:
address: :80
http:
redirections:
entryPoint:
to: websecure
scheme: https
permanent: true
websecure:
address: :443
http:
tls:
certResolver: le
forwardedHeaders:
trustedIPs:
- ***
ping:
entryPoint: websecure
providers:
docker:
exposedByDefault: false
endpoint: "unix:///var/run/docker.sock"
file:
directory: /etc/traefik
watch: true
experimental:
plugins:
real-ip:
moduleName: github.com/Paxxs/traefik-get-real-ip
version: v1.0.3
dynamic.yml
http:
middlewares:
test1:
chain:
middlewares:
- compression
- real-ip-plugin
test2:
retry:
attempts: 1
initialinterval: 100ms
redirect-to-www:
redirectregex:
regex: ^https://(?:www\.)?(.*)
replacement: https://www.$${1}
auth:
basicAuth:
users:
- "***"
compression:
compress:
excludedContentTypes:
- text/event-stream
security:
headers:
accessControlAllowOriginList:
- "*"
real-ip-plugin:
plugin:
real-ip:
Proxy:
- proxyHeadername: ***
proxyHeadervalue: ***
realIP: ***
OverwriteXFF: true
- proxyHeadername: "*"
realIP: ***
serversTransports:
insecure:
insecureSkipVerify: true
docker-compose.yml
networks:
traefik:
name: traefik
enable_ipv6: false
driver: bridge
ipam:
driver: default
config:
- subnet: 192.168.3.0/24
gateway: 192.168.3.1
services:
traefik:
#image: traefik:v2.11
image: traefik:latest
container_name: traefik
restart: unless-stopped
networks:
- traefik
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./config/:/etc/traefik/
- ${DOCKER_DATA}/traefik/acme:/acme
- ${DOCKER_DATA}/traefik/logs:/var/log/traefik
environment:
BASE_DOMAIN: ${BASE_DOMAIN}
CF_ZONE_API_TOKEN: ${CF_ZONE_API_TOKEN}
CLOUDFLARE_DNS_API_TOKEN: ${CF_ZONE_API_TOKEN}
labels:
- "traefik.docker.network=traefik"
- "traefik.enable=true"
- traefik.http.routers.traefik.rule=Host(`traefik.${BASE_DOMAIN}`)
- traefik.http.routers.traefik.entrypoints=websecure
- traefik.http.routers.traefik.service=api@internal
# Forward auth
- traefik.http.middlewares.auth-traefik.forwardauth.address=http://authentik-outpost:9000/outpost.goauthentik.io/auth/traefik
- traefik.http.middlewares.auth-traefik.forwardauth.trustForwardHeader=true
- traefik.http.routers.traefik.middlewares=security@file, test1@file, auth-traefik@docker
# Route logout pages
- traefik.http.routers.traefik-auth.middlewares=security@file, test1@file
- traefik.http.routers.traefik-auth.rule=Host(`traefik.${BASE_DOMAIN}`) && PathPrefix(`/outpost.goauthentik.io/`)
- traefik.http.routers.traefik-auth.service=authentik-proxy@docker
- traefik.http.routers.traefik-auth.entrypoints=websecure
- traefik.http.routers.traefik-auth.tls.domains[0].main=${BASE_DOMAIN}
- traefik.http.routers.traefik-auth.tls.domains[0].sans=*.${BASE_DOMAIN}
- com.centurylinklabs.watchtower.enable=true
- com.centurylinklabs.watchtower.scope=daily
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 11
- Comments: 53 (14 by maintainers)
Same here; I had the same issue after upgrade from 2.11 to 3.2, memory usage reached up to >5 GB, the nodes were acting crazy, how did they miss this issue with zstd !
I set this to fix it for now as I can’t rollback to 2.11 especially after the CRD api changes.
apiVersion: traefik.io/v1alpha1 kind: Middleware metadata: name: compress namespace: ****** spec: compress: defaultEncoding: gzip encodings: - gzip
I understand the reasoning, but if if I was a new Traefik user and it would use 5GB of RAM, I would no longer be a Traefik user.
V3 at the state of the 3.0 release would absolutely scare me away from it.
After running some tests, I can confirm that the memory usage is linked to the compression algorithms.
Here are the results I observed:
Without compression: Memory usage is around 2-3 MB. With Gzip enabled: Memory usage goes up to 20-30 MB. With Zstandard (Zstd) enabled: Memory usage is between 60-80 MB. With Brotli enabled: Memory usage increases to 350-500 MB. The new compression methods introduced in Traefik 3.x seem to require more memory compared to Gzip or no compression. Starting from version 3.2, there is a new feature that allows excluding certain compressions, which can help reduce memory usage. https://doc.traefik.io/traefik/v3.2/middlewares/http/compress/#encodings
I suggest testing with the 3.2 RC version, as the final release should arrive soon.
@deyanp @Janhouse I also had a memory leak with the v3 version, when I removed the compress middleware the memory leak disappeared. Maybe you can try it.
Closed by #11641.
There is still a memory issue with the zstd compression, not sure this issue can really be marked as solved.
I think the default option (zstd) having this issue wouldn’t let this be “solved”. Probably should revert the default to gzip.
I’d prefer to use zstd where possible since it yields slightly better performance/compression.
While I agree manually becoming aware of this issue, and having the knowledge to switch back to gzip, does resolve the symptoms, I have to consider the rock solid stability of 2.11 – I can’t risk encountering further potential tripwires within 3.x
@mattbha With v3.2 we recommend to configure the encodings you need thanks to the encodings option.
I am no longer experiencing OOM issues.
After testing v3.2 with gzip only, I confirm the issue is solved.
I’m using traefik v3.1.2 and I’m experiencing the same issue with a high memory consumption. No issue in traefik v2.
We also resolved it by switching the compression to gzip. As we have millions of requests per minute, our Traefik lasted for 2-3 minutes before hitting the 300 MB memory limit.
Looks like you should include stress testing so you can catch things like these before switching something to default. And thanks for the solution.
We disabled zstd compression in Traefik v3 after we run into memory leak issues.
After upgrading from v2 to v3, memory usage was six times higher than before, so our pods were OOM killed. We gave them more memory and it was fully utilized. After switching back to gzip, everything was back to normal.
Under a simple 100 user locust stress test I notice my traefik RAM usage (in a docker-compose stack) shot up to about 1.4gb before I stopped it
image: traefik:3.2When I changed the docker-compose to
image: traefik:2.11I hit about 140mb maxdefault settings everywhere, compression middleware enabled.
3.2 was choosing to use zstd 2.11 is using gzip of course
I cannot upgrade to 3.X from 2.11 due to this issue. Definitely appeared like a memory leak would (3.2 memory kept growing and growing under stress test, likely due to zstd).
Combined with the above problem analysis and solution, finally through the constraint of “preferential use of Gzip compression algorithm”, the same result is obtained in v3.2.0 and v2.11.13. Thank you very much!
@Janhouse I understand your concern, but compression is not enabled by default. So, a new Traefik user won’t immediately use 5GB of RAM unless they enable the compression middleware. That said, this is just my personal opinion. I’ll share your feedback with the other maintainers during the next triage and get back to you as soon as possible.
@Janhouse I wasn’t able to reproduce the leak, but I did reproduce the “larger memory footprint.”
Personally, if we change this now, it might feel like a breaking change. Some users may have higher memory limits and prefer to use more memory with all the compression enabled.
For me, the main issue was that we couldn’t configure it, but this will be possible in v3.2.
Hello, it’s currently my server configuration https://github.com/toutvukantabu/Homelab-traefik-authentik-crowdsec-server thanks a lot for your investigations!
Thank you all for your input and for sharing your experiences regarding the memory leak. After investigating on my end, I haven’t been able to reproduce the memory leak issue with any compression method, including br and zstd. While it’s true that compression does use additional memory, in my tests, memory usage returns to normal levels once traffic decreases.
However, there is one notable difference I would like to highlight between Traefik v2 and v3 regarding compression. In v2, only gzip was supported, whereas in v3, we added support for Brotli (br) and Zstandard (zstd). These newer compression algorithms are more memory-intensive compared to gzip. From my tests, I’ve observed that gzip consumes around 200MB, whereas Brotli and Zstandard can go up to 900MB.
To address this, a PR has been merged that introduces the ability to select which compression algorithms you want to enable. This feature will be available in the next minor release, which is expected to be released very soon.
In the meantime, to further investigate this potential memory leak, could someone provide a complete reproducible case? This would greatly help us narrow down the issue. Ideally, here’s what we would need:
The more detailed the scenario, the quicker we can identify and resolve the issue.
I’ve been experiencing the same issue after upgrading to v3. Memory usage would just keep increasing until an OOM event. I found that when i set
watch: falseon the file provider, however, the problem went away. I’m not sure why exactly, as I haven’t had a chance to investigate further, but perhaps try setting that to false and see if it helps?