thanos: High CPU/memory usage on querier/query frontend after update from 0.38.0 to 0.39.0

Thanos version used:

Thanos: 0.39.0 and 0.38.0

Object Storage Provider:

S3

What happened:

We are seeing an elevated use of CPU/memory when the query and queryFrontend are idle, after the update to 0.39.0. We see this occuring in multiple clusters.

What you expected to happen:

Memory/cpu consumption staying the same (or even improving, according to the changelogs in 0.39.0)

How to reproduce it (as minimally and precisely as possible):

Update from 0.38.0 to 0.39.0, or more specifically, update the bitnami thanos chart from 17.0.2 to 17.0.4

Full logs to relevant components:

Doesn’t seem to be relevant, but here are the query frontend logs: (0.39.0)

2025-06-26 08:07:37.951Z ts=2025-06-26T08:07:37.951843972Z caller=query_frontend.go:448 level=info msg="starting query frontend"
2025-06-26 08:07:37.952Z ts=2025-06-26T08:07:37.952102368Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
2025-06-26 08:07:37.952Z ts=2025-06-26T08:07:37.952121939Z caller=http.go:72 level=info service=http/server component=query-frontend msg="listening for requests and metrics" address=0.0.0.0:9090
2025-06-26 08:07:37.952Z ts=2025-06-26T08:07:37.952507439Z caller=handler.go:87 level=info service=http/server component=query-frontend caller=tls_config.go:347 time=2025-06-26T08:07:37.952498579Z msg="Listening on" address=[::]:9090
2025-06-26 08:07:37.952Z ts=2025-06-26T08:07:37.95255277Z caller=handler.go:87 level=info service=http/server component=query-frontend caller=tls_config.go:350 time=2025-06-26T08:07:37.95254589Z msg="TLS is disabled." http2=false address=[::]:9090
2025-06-26 08:07:37.955Z ts=2025-06-26T08:07:37.955481096Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready

And the 0.38.0 logs:

2025-06-10 08:04:31.736Z ts=2025-06-10T08:04:31.736584754Z caller=query_frontend.go:387 level=info msg="starting query frontend"
2025-06-10 08:04:31.736Z ts=2025-06-10T08:04:31.73666675Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
2025-06-10 08:04:31.736Z ts=2025-06-10T08:04:31.736840413Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
2025-06-10 08:04:31.736Z ts=2025-06-10T08:04:31.736860404Z caller=http.go:74 level=info service=http/server component=query-frontend msg="listening for requests and metrics" address=0.0.0.0:9090
2025-06-10 08:04:31.737Z ts=2025-06-10T08:04:31.737395773Z caller=handler.go:83 level=info service=http/server component=query-frontend caller=tls_config.go:347 time=2025-06-10T08:04:31.737387032Z msg="Listening on" address=[::]:9090
2025-06-10 08:04:31.737Z ts=2025-06-10T08:04:31.737472879Z caller=handler.go:83 level=info service=http/server component=query-frontend caller=tls_config.go:350 time=2025-06-10T08:04:31.737466148Z msg="TLS is disabled." http2=false address=[::]:9090

Anything else we need to know:

We use EKS and the bitnami helm chart to deploy thanos. If needed, more config can be provided. Here is a sample of usages: (pods at the bottom are 0.38.0, above are 0.39.0, same config). Image

Memory profiles:

0.39.0: heap-files.zip 0.38.0: heap-0.38-files.zip

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 18
  • Comments: 18 (7 by maintainers)

Most upvoted comments

Sorry for the problems, everyone. I also tried to look into the querier dumps but I cannot see anything odd there. I will close this issue to keep it focused on QFE. Could you please open another one with querier dumps if it shows odd resource consumption? Thank you!

#8349 could someone try QFE with this change?

So far (from 2 minutes of testing) memory and cpu usage is back to normal (about 12MB). 👍