traefik: Memory issues in v2 where memory never goes down even without traffic
Welcome!
- Yes, I’ve searched similar issues on GitHub and didn’t find any.
- Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.
What did you do?
We are running traefik 2.11.4 on our kubernetes clusters as the ingress controller. And we noticed that the memory usage of the traefik pod kept going up during the day and never went down even at night when traffic is mostly inexistant.
This seamed to be manageable at first but with the increase in our traffic, it lead to many OOMKilled during the day.
As of today we increased the memory even more and used GOMEMLIMIT to try to force the memory to go down more. But we still cannot explain the very high memory usage at night when we have very little traffic.
What did you see instead?
After a restart when there is no traffic, at night, the ram is low, but it goes up proportionally during the day and never goes down.
What version of Traefik are you using?
Version: 2.11.4
Codename: mimolette
Go version: go1.22.4
Built: 2024-06-10T13:20:58Z
OS/Arch: linux/amd64
What is your environment & configuration?
--global.checknewversion
--global.sendanonymoususage
--entrypoints.metrics.address=:9100/tcp
--entrypoints.traefik.address=:9000/tcp
--entrypoints.web.address=:8000/tcp
--entrypoints.websecure.address=:8443/tcp
--api.dashboard=true
--ping=true
--metrics.prometheus=true
--metrics.prometheus.entrypoint=metrics
--providers.kubernetescrd
--providers.kubernetesingress
--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik-prod-2
--entrypoints.websecure.http.tls=true
--api.debug=true
--api.insecure=true
--log.level=WARN
--log.format=json
--accesslog=true
--accesslog.format=json
--providers.file.filename=/config/dynamic.yaml
--entrypoints.web.http.redirections.entryPoint.to=:443
--entryPoints.web.forwardedHeaders.trustedIPs=<REDACTED>
--entryPoints.websecure.forwardedHeaders.trustedIPs=<REDACTED>
--entrypoints.web.http.redirections.entryPoint.scheme=https
--entrypoints.web.http.redirections.entryPoint.priority=10000
--entrypoints.websecure.transport.lifeCycle.requestAcceptGraceTimeout=2
--entrypoints.web.transport.lifeCycle.requestAcceptGraceTimeout=2
--accessLog.fields.defaultMode=keep
--accessLog.fields.headers.defaultMode=keep
--providers.kubernetesingress.allowexternalnameservices=true
--providers.kubernetescrd.allowexternalnameservices=true
--providers.kubernetescrd.allowCrossNamespace=true
--providers.kubernetesingress.allowEmptyServices=true
--experimental.localPlugins.authenticationATRT.moduleName=github.com/padoa/traefik-authentication-at-rt
--experimental.localPlugins.ondemand.moduleName=github.com/acouvreur/traefik-ondemand-plugin
--experimental.localPlugins.blockpath.moduleName=github.com/traefik/plugin-blockpath
We do use some middlewares, mainly the default ones like IPAllowList but also some community and custom ones that rely on Yaegi.
Finally, we do have a lot of ingress, and routers currently: 5000 routers and 150 services on our biggest cluster.
If applicable, please paste the log output in DEBUG level
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (4 by maintainers)
Can definitely confirm that
.19fixed my memory problem.Thanks for this comment, we started seeing strange issues after upgrading to 2.11.18, and once I saw the graph below, I downgraded to 2.11.16:
FWIW, I had similar symptoms—Traefik 2.11.18’s memory usage growing without bound—and setting
keepAliveMaxTimedidn’t seem to help. This was on a much smaller setup, not in K8s.Thanks for the suggestions, I will try this asap.
Hello @baptman21,
Thanks for reporting this!
At first glance, I would try to see if there is any improvement by limiting the time to live for connections, with either keepalivemaxrequests or keepAliveMaxTime. Could you please try either of these options?