thanos: Receive : Receive 0.36.1 in router / ingestor mode has a memory leak and goes OOM every 2 hrs

Thanos, Prometheus and Golang version used: Thanos : 0.36.1 Prometheus 2.54.1

/ $ thanos --version thanos, version 0.36.1 (branch: HEAD, revision: 99a5742a15f107d4607d280c825eca5b7f09a253) build user: root@3b4bc83e1037 build date: 20240813-11:33:32 go version: go1.21.13 platform: linux/amd64 tags: netgo

Object Storage Provider: Azure Blob Storage

What happened: Memory leak for Thanos receive in router ingestor mode

What you expected to happen:

No memory leaks happen

How to reproduce it (as minimally and precisely as possible): Not sure.

Full logs to relevant components: image

Anything else we need to know:

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 26 (10 by maintainers)

Most upvoted comments

@MichaHoffmann @yeya24 Thank you for your time. Very much appreciated ❤️

@MichaHoffmann Good news. After turning exemplar off, the memory is very very stable at 1.6 GB 😱

Do you have any idea on how often exemplar is flushed from memory?

image