strimzi-kafka-operator: [Bug]: KafkaRebalance resource reconciliation gets stuck if CC pod is restarted during rebalance
Bug Description
Encountered a weird behaviour in 0.40. If the CC pod is restarted in the middle of a rebalance, the KafkaRebalance resource reconciliation gets stuck with the following cluster operator logs:
2024-05-09 09:27:54 INFO KafkaRebalanceAssemblyOperator:1018 - Reconciliation #184(kafkarebalance-watch) KafkaRebalance(strimzi/my-rebalance): Getting Cruise Control rebalance user task status
2024-05-09 09:27:54 ERROR ContextImpl: - Unhandled exception
java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:361) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:427) ~[?:?]
at io.vertx.core.json.JsonArray.getJsonObject(JsonArray.java:243) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.strimzi.operator.cluster.operator.resource.cruisecontrol.CruiseControlApiImpl.lambda$getUserTaskStatus$14(CruiseControlApiImpl.java:328) ~[io.strimzi.cluster-operator-0.40.0.1.0.0-b274.jar:0.40.0.1.0.0-b274]
at io.vertx.core.impl.future.FutureImpl$1.onSuccess(FutureImpl.java:91) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:60) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:305) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:295) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:57) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:246) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:79) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.http.impl.HttpClientResponseImpl.handleEnd(HttpClientResponseImpl.java:250) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.lambda$new$0(Http1xClientConnection.java:418) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:255) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:134) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.http.impl.Http1xClientConnection$StreamImpl.handleEnd(Http1xClientConnection.java:705) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:320) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:300) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.http.impl.Http1xClientConnection.handleResponseEnd(Http1xClientConnection.java:941) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.http.impl.Http1xClientConnection.handleHttpMessage(Http1xClientConnection.java:811) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.http.impl.Http1xClientConnection.handleMessage(Http1xClientConnection.java:775) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:159) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:153) ~[io.vertx.vertx-core-4.5.4.jar:4.5.4]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty.netty-codec-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty.netty-codec-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) ~[io.netty.netty-handler-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[io.netty.netty-handler-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[io.netty.netty-handler-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[io.netty.netty-codec-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[io.netty.netty-codec-4.1.107.Final.jar:4.1.107.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[io.netty.netty-codec-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.107.Final.jar:4.1.107.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.107.Final.jar:4.1.107.Final]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
Steps to reproduce
- Create a kafka cluster with 1 nodepool, 3 brokers and CC
- Create an imbalanced state (e.g. 1 topic, 120 partitions, RF=2 and all replicas on broker 0 and 1) (Just make sure that rebalancing from this state would take at least a few minutes)
- Trigger a rebalance
- After the rebalance started, scale up the nodepool to 4
- Observe as the CC pod gets restarted due to the new broker in the middle of the rebalance
- Observe how the reconciliation of the KafkaRebalance resource gets stuck (CC does not persist tasks)
Expected behavior
No response
Strimzi version
0.40.0
Kubernetes version
v1.28.7+k3s1
Installation method
helm
Infrastructure
Bare-metal
Configuration files and logs
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
name: pool-a
labels:
strimzi.io/cluster: my-cluster
spec:
replicas: 3
roles:
- broker
storage:
type: ephemeral
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
annotations:
strimzi.io/node-pools: enabled
spec:
kafka:
version: 3.7.0
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: "3.7"
zookeeper:
replicas: 3
storage:
type: ephemeral
entityOperator:
topicOperator: {}
userOperator: {}
cruiseControl: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
name: my-rebalance
labels:
strimzi.io/cluster: my-cluster
# no goals specified, using the default goals from the Cruise Control configuration
spec: {}
Additional context
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 29 (29 by maintainers)
I think that if the original proposal was reviewed and manually triggered, an automatic new proposal + trigger is counter-intuitive. Especially if the proposal is significantly different than the previously reviewed one. For auto-approved proposals, the automatic trigger sounds reasonable.
@ShubhamRwt Please start a discussion on Slack in #strimzi-dev on this. That should be better than doing chat on the issue. I cannot do that as I do not know which account is actually you. 😉
Triaged on 30/5/2024: it seems to be a bug which needs to be investigated and reproduced first. @ShubhamRwt let’s get in touch to understand how you tried to reproduce it with no success and what we can do.
Thanks, fixed the formatting.