traefik: Memory issues in v2 where memory never goes down even without traffic

Welcome!

  • Yes, I’ve searched similar issues on GitHub and didn’t find any.
  • Yes, I’ve searched similar issues on the Traefik community forum and didn’t find any.

What did you do?

We are running traefik 2.11.4 on our kubernetes clusters as the ingress controller. And we noticed that the memory usage of the traefik pod kept going up during the day and never went down even at night when traffic is mostly inexistant.

This seamed to be manageable at first but with the increase in our traffic, it lead to many OOMKilled during the day.

As of today we increased the memory even more and used GOMEMLIMIT to try to force the memory to go down more. But we still cannot explain the very high memory usage at night when we have very little traffic.

What did you see instead?

After a restart when there is no traffic, at night, the ram is low, but it goes up proportionally during the day and never goes down.

Image

What version of Traefik are you using?

Version:      2.11.4
Codename:     mimolette
Go version:   go1.22.4
Built:        2024-06-10T13:20:58Z
OS/Arch:      linux/amd64

What is your environment & configuration?

--global.checknewversion
--global.sendanonymoususage
--entrypoints.metrics.address=:9100/tcp
--entrypoints.traefik.address=:9000/tcp
--entrypoints.web.address=:8000/tcp
--entrypoints.websecure.address=:8443/tcp
--api.dashboard=true
--ping=true
--metrics.prometheus=true
--metrics.prometheus.entrypoint=metrics
--providers.kubernetescrd
--providers.kubernetesingress
--providers.kubernetesingress.ingressendpoint.publishedservice=traefik/traefik-prod-2
--entrypoints.websecure.http.tls=true
--api.debug=true
--api.insecure=true
--log.level=WARN
--log.format=json
--accesslog=true
--accesslog.format=json
--providers.file.filename=/config/dynamic.yaml
--entrypoints.web.http.redirections.entryPoint.to=:443
--entryPoints.web.forwardedHeaders.trustedIPs=<REDACTED>
--entryPoints.websecure.forwardedHeaders.trustedIPs=<REDACTED>
--entrypoints.web.http.redirections.entryPoint.scheme=https
--entrypoints.web.http.redirections.entryPoint.priority=10000
--entrypoints.websecure.transport.lifeCycle.requestAcceptGraceTimeout=2
--entrypoints.web.transport.lifeCycle.requestAcceptGraceTimeout=2
--accessLog.fields.defaultMode=keep
--accessLog.fields.headers.defaultMode=keep
--providers.kubernetesingress.allowexternalnameservices=true
--providers.kubernetescrd.allowexternalnameservices=true
--providers.kubernetescrd.allowCrossNamespace=true
--providers.kubernetesingress.allowEmptyServices=true
--experimental.localPlugins.authenticationATRT.moduleName=github.com/padoa/traefik-authentication-at-rt
--experimental.localPlugins.ondemand.moduleName=github.com/acouvreur/traefik-ondemand-plugin
--experimental.localPlugins.blockpath.moduleName=github.com/traefik/plugin-blockpath

We do use some middlewares, mainly the default ones like IPAllowList but also some community and custom ones that rely on Yaegi.

Finally, we do have a lot of ingress, and routers currently: 5000 routers and 150 services on our biggest cluster.

Image

If applicable, please paste the log output in DEBUG level

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (4 by maintainers)

Most upvoted comments

Image

Can definitely confirm that .19 fixed my memory problem.

FWIW, I had similar symptoms—Traefik 2.11.18’s memory usage growing without bound—and setting keepAliveMaxTime didn’t seem to help. This was on a much smaller setup, not in K8s.

Thanks for this comment, we started seeing strange issues after upgrading to 2.11.18, and once I saw the graph below, I downgraded to 2.11.16:

Image

FWIW, I had similar symptoms—Traefik 2.11.18’s memory usage growing without bound—and setting keepAliveMaxTime didn’t seem to help. This was on a much smaller setup, not in K8s.

Thanks for the suggestions, I will try this asap.

Hello @baptman21,

Thanks for reporting this!

Possibly stupid has I don’t know the code of traefik well but, since there is many config reloading, active connections could keep the old config used and prevent it to be GCed, resulting in the memory usage going up. However once again it this is the case, why isn’t this memory visible in pprof ?

At first glance, I would try to see if there is any improvement by limiting the time to live for connections, with either keepalivemaxrequests or keepAliveMaxTime. Could you please try either of these options?