logstash-input-kafka: topics_pattern not working with logstash 5.4.0 and kafka-input 5.1.7

I’ve setup the following, simple pipeline:

input {
  kafka {
    bootstrap_servers => "localhost:9092"
    topics_pattern => "aaa.bbb_ccc.*"
  }
}

output {
  stdout { }
}

and logstash starts successfully with no issues written to the log in debug mode. However, it doesn’t consume any message from the topic (new or pre-existing), while kafkacat started in parallel is consuming the events from the same broker and topic correctly.

Has anyone ever managed run a working example using topics_pattern? I’m trying to read from multiple topics fed through an environment variable, so it doesn’t make sense to define an input-per-topic or use the topics => ["aaa.bbb_ccc.1", aaa.bbb_ccc.2" ... "aaa.bbb_ccc.n"] syntax instead.

Any ideas what might be missing or wrong with this usage pattern?

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 1
Comments: 17 (7 by maintainers)

Most upvoted comments

@harelmoshe @GoodMirek

I just tried this out with the most recent version of this plugin 6.3.2 and it works for me as long as I start Logstash with the topics already existing in Kafka.

For creating topics and having them be subscribed to right away you will need to lower the setting for metadata_max_age_ms, the default here is 300000 == 5 minutes. After those 5 minutes, the new topic should be subscribed to just fine as well in the default configuration.

original-brownbear on Jul 7, 2017