kafkajs: ERROR: The group is rebalancing, so a rejoin is needed
Hi,
i need some help with the error message from the title. I read this issue, which seems to be related:
https://github.com/tulios/kafkajs/issues/226
I also read the documentation here, which also seems to be related: https://kafka.js.org/docs/faq#what-does-it-mean-to-get-rebalance_in_progress-errors
However, it doesnt matter what i try the errors are remaining. It only occures when i redeploy my application and unfortunately it always triggers our alerting system, which is annoying.
Here is the log from my application:
{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:32:41.900Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":16,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:32:41.901Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":3560}"}
{"loglevel":"error","msg":"ERROR [Connection] Response SyncGroup(key: 14, version: 1) {\"timestamp\":\"2020-04-06T10:33:41.906Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":18,\"size\":14}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:35:41.919Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-0b9827ef-9e82-4343-96d7-3bee2053c449\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[11,15,19,23,27,3,31,35,39,7],\"my-topic-2\":[11,15,19,23,27,3,31,35,39,7]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":180018}"}
{"loglevel":"error","msg":"ERROR [Connection] Response Heartbeat(key: 12, version: 1) {\"timestamp\":\"2020-04-06T10:37:17.024Z\",\"logger\":\"kafkajs\",\"broker\":\"kafka-2:9093\",\"clientId\":\"my-custom-client-id-server\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"correlationId\":45,\"size\":10}"}
{"loglevel":"error","msg":"ERROR [Runner] The group is rebalancing, re-joining {\"timestamp\":\"2020-04-06T10:37:17.025Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"error\":\"The group is rebalancing, so a rejoin is needed\",\"retryCount\":0,\"retryTime\":2618}"}
{"loglevel":"info","msg":"INFO [Runner] Consumer has joined the group {\"timestamp\":\"2020-04-06T10:37:41.916Z\",\"logger\":\"kafkajs\",\"groupId\":\"my-custom-client-id-v001-server\",\"memberId\":\"my-custom-client-id-server-bea7349c-32f2-4855-92a2-c2549f172082\",\"leaderId\":\"my-custom-client-id-server-69d3bcd8-a1d9-428f-8742-142ccb674b67\",\"isLeader\":false,\"memberAssignment\":{\"my-topic-1\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9],\"my-topic-2\":[1,11,13,15,17,19,21,23,25,27,29,3,31,33,35,37,39,5,7,9]},\"groupProtocol\":\"RoundRobinAssigner\",\"duration\":24891}"}
I already tried to increase the following config parameters.
maxRetryTime,sessionTimeout, rebalanceTimeout,heartbeatInterval
Neither helped, any hints would be greatly apprechiated.
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 19
- Comments: 15
I too believe that rebalancing should not be logged as error given that it is part of the normal protocol/workflow for kafka when a consumer joins/leaves a consumer group. No other libraries for kafka log this as an error ( springboot, kafka-node,…) . I believe this should be at worst logged as a warning if not a info.
At infrastructure/devops world, an error means an alert that could ring someone’s phone at night… So if its a “normal error” that is not urgent, means it should be a warning, not an error.
+1 on changing that from ERR to WARN
Thanks!
Can we reopen the issue? @cristianbriscaru argument is pretty solid.
The thing is, they are errors, which is why they are logged as such. It’s just that sometimes they are expected errors. However, only you have the context to determine when it is fine (during deploys) and when it isn’t. I would suggest that you adjust your alerting to not alert during deploys.
If this is not possible for whatever reason, such as if there’s no way for your monitoring system to know that there is a deploy ongoing, then one workaround I can think of is to use your own custom logger that specifically looks for “The group is rebalancing, re-joining” and logs it as a warn rather than an error. I don’t recommend it, but it is a possible solution.
Can we reopen this issue and turn
[Connection] Response Heartbeatinto a warning as well?Is there any solution for this solution? I am getting this error too.
Since the log message was changed from ERROR to WARN in one place, can it be fixed in another?
Curious, is there a justification for not logging
[Connection] Response Heartbeatat WARN level too?Really dislike the idea of having to implement a custom logger to catch these and change the logging level.
It looks like #1279 fixed that error:
while there’s remaining one:
I agree at some extent with @Nevon that alerts probably shouldn’t be based on error logs, but at the same time I don’t agree that error logs should be part of expected flow.
@Nevon
In the middle of data processing in consumer , If Consumer try to Rebalance , Then my existing data may not process correctly as it is going to join in new group.
My questions are
#1279 now downgrades this log to WARN. See https://github.com/tulios/kafkajs/pull/1279#issuecomment-1028796625 for my rationale for accepting the change.
For reference, I myself work as a devops engineer and work with infrastructure and observability on a daily basis, providing a deployment platform for thousands of services, many of which are using KafkaJS. The reason this has not been a problem within our organization is that we base our observability strategy on metrics rather than logs, and generally alert based on SLOs. With such a strategy, it doesn’t matter if the consumer rebalances as long as it doesn’t affect the SLIs to the point where you will exhaust your error budget. Symptom-based alerting can still be achieved by emitting metrics from instrumentation events. This is the background for why I’ve been adamant to not make this change, as I think alerting on logs is an anti-pattern that this change now encourages.
However, I’ll just disagree and commit on this one, since clearly many people are doing this today and don’t want to control the log level themselves via a custom logger.
As the documentation suggests, this is a normal occurrence during deploys:
Whenever a member leaves or joins the consumer group (such as when you redeploy), a rebalance is needed, in order to assign work across all members of the consumer group. The members become aware of this when the broker responds to an API call with an error indicating that they need to resync.
During deploys, you are adding and removing nodes from the group, which means that the group cannot continue processing messages during that time (or at least they will be frequently interrupted when they have to stop and resync), so it is correct that your application isn’t actually doing its job during that time. On the other hand, if the issue persist after your deployment is over, then it means that your application still isn’t working and should probably trigger an alert.
Take a look at the documentation for these parameters to understand further what they do.
maxRetryTimeandheartbeatIntervalhave nothing to do with this, and changing their values without understanding what they do could have bad consequences. In particular increasingheartbeatIntervalcould mean that the application heartbeats too infrequently and may cause the broker to assume that it is dead and kick it out of the group (depending on the value ofsessionTimeout.