node-rdkafka: Memory leak on disconnect() method usage with producer

Hi,

In one of our application we are using nodejs based microservice which internally uses node-rdkafka module to send messages using kafka producer which involves making runtime call scheduled at specific interval to perform below operations:

Invoking kafka producer connect method
Calling producer using kafka produce method to send messages
Invoking kafka producer disconnect method

Disconnect method usage:

let prod =  new Kafka.Producer({
// config properties
})
prod.disconnect()

However, we have been noticing memory leak when calling node-rdfafka producer and suspecting that it is disconnect method which is causing the leak.

Thanks!

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (2 by maintainers)

Commits related to this issue

Disable node-rdkafka event_cb if not using kafkaEventHandlers This is a workaround for a memory leak bug currently in node-rdkafka: https://github.com/Blizzard/node-rdkafka/issues/731 If you do set a... — committed to wikimedia/KafkaSSE by ottomata 4 years ago

Most upvoted comments

I’ve pushed preliminary fix on fix-producer-leaks branch. I still need help with testing. It would be great if some of you can try it out and see if it helps to prevent Producer memory leaks and also check if events are still flowing in as expected.

Also, it seems that Consumer has leaks when rebalance_cb or offset_commit_cb are set and these need to be addressed as well.

iradul on May 20, 2020

Hi @iradul, I just tried out your fix and it looks like the memory leak in HighLevelProducer is gone now! The memory allocation timeline of our service running with your fix shows a much cleaner heap.

In comparison, this is the memory allocation timeline of our service running on the latest master.

memory-allocation-master

The active objects allocated in the middle of the master graph are HighLevelProducer instances which should have already been garbage collected by the end of the recording

ArneSchulze on May 26, 2020

Setting event_cb = false does indeed fix the memory leak.

ottomata on Mar 23, 2020

I think that’s pretty much the definition of “often”!

Each connection is long lived and unending. The HTTP response body streamed to the client in chunked-transfer encoding. We have about 60-80 concurrent connections, with about what currently looks like 6-10 reconnects per minute (which still seems like a lot to me, likely some remote client is not doing it right :p).

Each consumer has specific subscription params (topics, offsets, etc.)

ottomata on Mar 19, 2020

@codeburke Yes we are creating multiple producer instances once for each runtime call (scheduled at specific time interval) we make which involves invoking connect() and disconnect() method at every call

Node version - 10
node-rdkafka - 2.7.4

Rajat-Sharma-2710 on Dec 20, 2019