logstash-input-kafka: Kafka logstash input do not continue from where it left off on restart

This is highly undesirable, the reason we are publishing data to kafka is to ensure the consumers can be taken down and can come back up in an asynchronous fashion. Is this a limitation of this plugin? or simply a configuration issue.

Also where does the kafka input save the kafka position per partition ?

I am using the latest version of the plugin with the following kafka config

input {
    kafka {
        zk_connect => "zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181"
        topic_id => "logs"
        consumer_threads => 1
        consumer_restart_on_error => true
        consumer_restart_sleep_ms => 100
        decorate_events => true
    type => "logs"
    }
}

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 58 (14 by maintainers)

Most upvoted comments

I have currently a case where the logstash kafka consumer is lagging behind. Now this happend twice this week, and everytime I restarted logstash kafka input the plugin continued from the latest point losing all the logs. Since I have the problem in my grip at the moment, I would like the folks here to tell me how I can check the logstash (without restarting) to help you debug what is going on.

@joekiller @suyograo screen shot 2016-02-07 at 10 04 03 pm