jaeger: jaeger has a memory leak when use badger

Describe the bug I’m doing a stress on badger and found that jaeger has a memory leak .

To Reproduce Steps to reproduce the behavior:

start jaeger with docker docker run --rm --name jaeger-badger -p 16831:6831/udp -p 16832:6832/udp -p 15778:5778 -p 15775:5775 -p 26686:16686 -p 24269:14269 -e SPAN_STORAGE_TYPE=badger -e BADGER_EPHEMERAL=false -e BADGER_DIRECTORY_VALUE=/badger/data -e BADGER_DIRECTORY_KEY=/badger/key jaegertracing/all-in-one:1.19 --badger.span-store-ttl 1m --badger.truncate

2.run a stress application to send data to jaeger. My application send 7000/s span data to jaeger with udp sender.

3.monitoring the memory of CONTAINER ;

Expected behavior jaeger’s memory should be stable at some point not always increase.

Screenshots If applicable, add screenshots to help explain your problem. as time goes on ,the memory become higher.

Version (please complete the following information):

OS: centos7
Jaeger version: 1.9
Deployment: docker

What troubleshooting steps did you try? I try to adjust some param. such as --collector.queue-size,--collector.queue-size-memory --badger.span-store-ttl ,--badger.consistency,--badger.maintenance-interval

Additional context

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 24 (12 by maintainers)

Most upvoted comments

Switching from 1x to 2x won’t reduce the memory. You’ll have to enable the index cache to see memory improvement.

You have to set the options.IndexCache size in badger. Wherever you’re opening badger, set the IndexCacheSize in badger options.

jarifibrahim on Sep 15, 2020

FWIW we have had success leaving the new Go memory behavior alone. It does obscure the amount of memory used by a process, but we have switched to using go_memstats_heap_alloc_bytes to watch for memory leaks. The new behavior is designed for better performance and responds well to linux memory pressure and cgroup limits.

joe-elliott on Sep 10, 2020