kafka-node: Does KafkaClient tolerate failure of broker(s) and try to connect other brokers ?
Questions?
Does KafkaClient tolerate failure of broker(s) and try to connect other brokers ?
Bug Report
Hi team, I am using KafkaClient as client parameter of Producer, and I set kafkaHost to “host1:port1,host2:port2,host3:port3” , it works when I send a message. Then, I kill the broker of host1:port1 and send message again via using the cached producer, it is stucked until request timeout, error message is TimeoutError: Request timed out after 10000ms. Then I use debug mode, and check the log of kafka-node, it always display kafka-node:KafkaClient kafka-node-client reconnecting to host1:port1… will it try to connect to host2:port2 or host3:port3 that I provided ? Also, when I debug in client.js, it has a method called getBrokers, seems that it doesn’t include all brokers I provided, sometimes only host1:port1, but sometimes [host1:port1, host2:port2] ?
Environment
- Node version: v6.5.9
- Kafka-node version: 2.3.0
- Kafka version: 1.0.0
- Scala version: 2.11
For specific cases also provide
- Number of Brokers: 3
- Number partitions for topic: 1
Include Sample Code to reproduce behavior
var client = new kafka.KafkaClient({
kafkaHost: "host1:port1,host2:port2,host3:port3",
connectTimeout: 10000,
requestTimeout: 10000
});
producer = new kafka.Producer(client);
Include output with Debug turned on
kafka-node:KafkaClient checking payload topic/partitions has leaders kafka-node:KafkaClient found leaders for all kafka-node:KafkaClient sending request kafka-node:KafkaClient grouped requests by 1 brokers [“0”] kafka-node:KafkaClient missing apiSupport waiting until broker is ready… kafka-node:KafkaClient kafka-node-client reconnecting to host1:port1 kafka-node:KafkaClient kafka-node-client reconnecting to host1:port1 kafka-node:KafkaClient kafka-node-client reconnecting to host1:port1 kafka-node:KafkaClient kafka-node-client reconnecting to host1:port1 … TimeoutError: Request timed out after 10000ms
Best Regards, Thanks!
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 15 (1 by maintainers)
We leave it up to the user of the module to determine when they should retry requests or give up. It’s difficult for the module to do this because it could make a wrong decision which could lead to messages being produced in the wrong order. Order can be very important for certain use cases.
oh I see, so it will timeout if the leader broker of topic is broken, and now I have add my own retry logic to send again if it timeout. I have suggestion: could KafkaClient add a option to retry if it timeout in case it caused by leader broker broken (new leader was elected and new topic/broker metadata can be fetch)?, so no exception will raised, it will not end up my process and I don’t need to add my own retry feature, if I get time, I will try to achieve it.
Thanks!
@hyperlink I think there might actually be a problem reconnecting on version 2.6.1.
Description
I created a
TestTopicwith 1 partition and replication factor 3.The producer produces a message at a regular interval. When I kill the leader, the producer isn’t able to recover, even in subsequent messages. From what I observed, the client didn’t round robin on the initial broker list to reconnect.
In the test consumer, when I kill two of the three brokers, it is unable to use the third broker to connect.
My Setup
Producer Debug Output
Consumer Debug Output
How did the client get the new broker metadata before sending the second request? KafkaClient only requests metadata if the connection to the broker is broken or if it doesn’t have metadata about a topic/partition it’s trying to fetch which shouldn’t be the case here since the first request went through. If you have more details please attach them.
In this case you will need to send the second request again. The issue here is the client’s idea of who is the leader of that topic/partition has changed so that second request is no longer valid. Once metadata has been updated the client is able to send the produce request to the correct broker. Regarding the timeout-- TCP sockets tries to be very forgiving and tries to keep the connection going for a long time before a signal is sent to node. We implemented our own timeout for this case and you can configure a lower timeout if you’d like.
Hi, debug again, and found something.
If I kill the leader broker of a topic, it will be stuck, and KafkaClient would NOT try to refresh metadata. If I kill the broker which is not the leader of this topic, it still works. Then I divide into the code, in KafkaClient.js, from line:789 to line:801
and in Client.js from line:598 to line:603
If I kill the leader of a topic, there still exist the leader broker in brokerMetadata of Client.js, so leaderLessPayloads.length === 0 is always satisfied, then it goes to reconnecting to the leader broker… I think kafka will elect a new leader of this topic if the old leader killed, isn’t it? but why the broker metadata is not refreshed if brokers changed ?
Thanks!