kminion: kafka-minion stuck on "Offsets topic has not yet been consumed until the end" for days
Hi, after succesfully testing kafka-minion locally and in a test environment, I deployed kafka-minion in a existing production environment with a long running Kafka cluster.
The test environment was created just for the purpose of evaluation kafka-minion, so there was not much data in the __consumer_offsets topic, and it started pretty much instantly.
In the production environment the Kakfa cluster has been running for months, so I expected that it would take some time for kafka-minion to start (in other issues people mentioned startup times in the 10 minute range), but in my case it is not ready after 3 days of running.
The logs over the last 3 days only show this:
kafka-minion[30248]: {"level":"info","msg":"Offets topic has not yet been consumed until the end","time":"2019-08-29T08:40:56Z"}
My question is what could cause kafka-minion to take so long to initialize, and what can I do to fix this?
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 39 (11 by maintainers)
With so many reports of people telling you that there are issue consuming __consumer_offsets topics can you still argue that this design choice is good ? at least you can provide the two methods and add a flag.
@weeco Interesting fact we just found in our investigation of Kafka minion. It’s seems like this problem only occurs when the __consumer_offsets topic is extremely unbalanced. we have a situation in one of our clusters that we have 108,958,210 messages for this topic (which has 50 partitions) and 108,482,731 of them are on the same partition. still investigating why but looks like this is the problem in our case
During the holidays between christmas and new year’s eve I’ll have some time to investigate and potentially rewrite the logic for that. I am eager to fix your reported problems. Any additional information might be helpful, as I can not reproduce it in any of my own clusters.
@weeco They already wrote a base 😃 But I’ll see if I can find something with kowl after I return from vacation. As for Burrow, we left it for the minion, because it didn’t provide lag per topic.
btw: noticed new release and tried it, no luck.