thanos: Receive : Receive 0.36.1 in router / ingestor mode has a memory leak and goes OOM every 2 hrs
Thanos, Prometheus and Golang version used: Thanos : 0.36.1 Prometheus 2.54.1
/ $ thanos --version thanos, version 0.36.1 (branch: HEAD, revision: 99a5742a15f107d4607d280c825eca5b7f09a253) build user: root@3b4bc83e1037 build date: 20240813-11:33:32 go version: go1.21.13 platform: linux/amd64 tags: netgo
Object Storage Provider: Azure Blob Storage
What happened: Memory leak for Thanos receive in router ingestor mode
What you expected to happen:
No memory leaks happen
How to reproduce it (as minimally and precisely as possible): Not sure.
Full logs to relevant components:
Anything else we need to know:
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 26 (10 by maintainers)
@MichaHoffmann @yeya24 Thank you for your time. Very much appreciated ❤️
@MichaHoffmann Good news. After turning exemplar off, the memory is very very stable at 1.6 GB 😱
Do you have any idea on how often exemplar is flushed from memory?