kminion: Fails to start if any seed broker is offline

This is part of the ongoing saga of figuring out what’s missing to make kminion able once more to operate with a kafka cluster where one or more brokers may be offline at any time.

Starting simple, here’s an immediate crash if any of the seed brokers are offline at startup:

{"level":"info","ts":"2021-06-01T06:30:58.438Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":"kafka1:9092,kafka2:9092,kafka3:9092"}
{"level":"warn","ts":"2021-06-01T06:31:01.521Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"fatal","ts":"2021-06-01T06:31:01.521Z","msg":"failed to test connectivity to Kafka cluster","error":"failed to request api versions: unable to dial: dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 25 (10 by maintainers)

Commits related to this issue

allow any *os.SyscallError to be retriable See embedded comment. Fixes cloudhut/kminion#93, as well as other issues. — committed to twmb/franz-go by twmb 3 years ago

Most upvoted comments

Let me include some fuller logs then. In this case, it’s the second broker in the list that is down (conveniently took these logs from the staging cluster where node 2 is currently down. Happy to test some other case if need be.)

{"level":"info","ts":"2021-06-07T08:09:42.835Z","msg":"started kminion","version":"ea47f264803f16dd2fdf3cae1d2b9ba7228ce4a9"}
{"level":"info","ts":"2021-06-07T08:09:42.835Z","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":"kafka-s101:9092,kafka-s201:9092,kafka-s301:9092"}
{"level":"warn","ts":"2021-06-07T08:09:50.897Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"info","ts":"2021-06-07T08:09:51.011Z","msg":"successfully connected to kafka cluster","advertised_broker_count":2,"topic_count":6,"controller_id":1003,"kafka_version":"v2.6"}
{"level":"info","ts":"2021-06-07T08:09:51.012Z","msg":"listening on address","listen_address":":8080"}
{"level":"warn","ts":"2021-06-07T08:09:59.057Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:02.129Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:05.201Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:08.273Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:11.345Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:14.421Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:17.489Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:20.561Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:23.633Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:26.705Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:29.777Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:32.849Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:35.921Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:38.993Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:42.065Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:45.137Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:48.209Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:51.281Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:54.353Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:10:57.429Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:00.497Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:03.569Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:06.641Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:06.641Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:06.641Z","msg":"broker failed to respond to the described groups request","broker_id":-2147483647,"error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:06.641Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"warn","ts":"2021-06-07T08:11:09.717Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:12.789Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:15.857Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:18.929Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:18.929Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:18.929Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:18.929Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"warn","ts":"2021-06-07T08:11:22.001Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:25.073Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:28.105Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:28.105Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:28.105Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:28.105Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp xxx.xxx.xxx.xxx:9092: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:28.105Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:28.105Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:28.145Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:31.217Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:34.289Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:37.361Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:37.361Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:37.361Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:37.361Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:37.361Z","msg":"broker failed to respond to the described groups request","broker_id":-2147483647,"error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:37.361Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:37.361Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:37.361Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:37.361Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:37.361Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:37.361Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:37.454Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"warn","ts":"2021-06-07T08:11:40.433Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:43.505Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:46.577Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:46.577Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:46.577Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:46.577Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:11:46.577Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:46.577Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:46.577Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:46.577Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:46.577Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:11:46.577Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:11:46.578Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"warn","ts":"2021-06-07T08:11:49.649Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:52.721Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:55.793Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:11:58.865Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:01.937Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:01.937Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:12:01.937Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:12:05.009Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:08.081Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:11.154Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:14.225Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:17.297Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:20.369Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:23.441Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:26.513Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:29.585Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:32.661Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:32.661Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:12:32.661Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:12:35.729Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:38.805Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:38.805Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:12:38.805Z","msg":"failed to get kafka metadata","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:12:38.805Z","msg":"failed to get kafka metadata","error":"failed to request metadata: context deadline exceeded"}
{"level":"warn","ts":"2021-06-07T08:12:41.873Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:44.945Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:48.017Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:51.089Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:12:54.161Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"error","ts":"2021-06-07T08:12:54.358Z","msg":"failed to describe a broker's log dirs","broker_address":"kafka-s101.dbc.dk","broker_id":"1001","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:12:54.358Z","msg":"failed to describe a broker's log dirs","broker_address":"kafka-s301.dbc.dk","broker_id":"1003","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:12:54.358Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:12:54.358Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:12:54.358Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"warn","ts":"2021-06-07T08:12:57.233Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:13:00.305Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:13:00.305Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:13:00.305Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:13:00.305Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:13:00.305Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to fetch low water marks","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to describe a broker's log dirs","broker_address":"","broker_id":"-1","error":"unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to fetch low water marks","error":"context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to collect consumer groups, because Kafka request failed","error":"failed to list consumer groups: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:00.305Z","msg":"failed to get topic configs","error":"failed to request metadata: context deadline exceeded"}
{"level":"warn","ts":"2021-06-07T08:13:03.377Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:13:06.453Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:13:09.525Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp xxx.xxx.xxx.xxx:9092: connect: no route to host"}
{"level":"warn","ts":"2021-06-07T08:13:09.525Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"warn","ts":"2021-06-07T08:13:09.525Z","msg":"unable to open connection to broker","source":"kafka_client","addr":"kafka-s201:9092","broker":-2147483647,"err":"dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:09.525Z","msg":"failed to fetch low water marks","error":"failed to list consumer groups: failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:09.525Z","msg":"failed to get topic configs","error":"failed to request metadata: unable to dial: dial tcp: i/o timeout"}
{"level":"error","ts":"2021-06-07T08:13:09.525Z","msg":"failed to fetch low water marks","error":"failed to list consumer groups: failed to request metadata: context deadline exceeded"}
{"level":"error","ts":"2021-06-07T08:13:09.525Z","msg":"failed to get metadata","error":"failed to request metadata: context deadline exceeded"}

srhb on Jun 7, 2021