strimzi-kafka-operator: [Bug]: TopicOperator removes throttled.replicas topic configuration of throttled reassignment/rebalance
Bug Description
When running kafka-reassign-partitions.sh with the --throttle flag or running a KafkaRebalance with replicationThrottle, Kafka will set these configurations on the topics that are currently moved: follower.replication.throttled.replicas: <currently moving replicas>, leader.replication.throttled.replicas: <currently moving replicas> (KIP-542).
The topic operator then picks up the configuration discrepancy and removes the configurations, which leads to unthrottled replica movements.
I would like to create a PR for this if you agree with the expected behavior.
Steps to reproduce
- Create cluster with 2 brokers, cruise control and enable
DEBUGlogging for topic operator - Create some test
KafkaTopic - Create
KafkaRebalancein which you remove a broker and setreplicationThrottle(see Additional Context) - Approve
KafkaRebalance - Wait until Cruise Control starts rebalance
- Observe in topic operator logs:
DEBUG [LoopRunnable-0] BatchingTopicController:987 - Reconciliation #7(upsert) KafkaTopic(kafka/rebalance-test): Config changes [AlterConfigOp{opType=DELETE, configEntry=ConfigEntry(name=follower.replication.throttled.replicas, value=null, source=UNKNOWN, isSensitive=false, isReadOnly=false, synonyms=[], type=UNKNOWN, documentation=null)}, AlterConfigOp{opType=DELETE, configEntry=ConfigEntry(name=leader.replication.throttled.replicas, value=null, source=UNKNOWN, isSensitive=false, isReadOnly=false, synonyms=[], type=UNKNOWN, documentation=null)}] - You can also check the topic configuration in Kafka to see that they are not set
Expected behavior
I guess the preferred behavior would be to ignore the topic configurations follower.replication.throttled.replicas and leader.replication.throttled.replicas if they are set on the broker, but not explicitly specified in the KafkaTopic resource spec.
If they are explicitly defined, the topic operator might be expected to overwrite them again.
Strimzi version
0.39.0, 0.40.0
Kubernetes version
v1.27.3
Installation method
Helm chart
Infrastructure
Kind, Bare-metal
Configuration files and logs
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 3.7.0
replicas: 2
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
config:
offsets.topic.replication.factor: 1
transaction.state.log.replication.factor: 1
transaction.state.log.min.isr: 1
default.replication.factor: 1
min.insync.replicas: 1
inter.broker.protocol.version: "3.7"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 1
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
topicOperator:
logging:
type: inline
loggers:
rootLogger.level: INFO
logger.top.name: io.strimzi.operator.topic
logger.top.level: DEBUG
logger.toc.name: io.strimzi.operator.topic.TopicOperator
logger.toc.level: TRACE
logger.clients.level: DEBUG
userOperator: {}
cruiseControl: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
name: rebalance-test
labels:
strimzi.io/cluster: my-cluster
spec:
topicName: rebalance-test
partitions: 100
Additional context
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
name: rebalance-test
labels:
strimzi.io/cluster: my-cluster
spec:
mode: remove-brokers
brokers: [1]
replicationThrottle: 10
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (14 by maintainers)
Commits related to this issue
- Fix TO-CC race condition issue happening during throttled rebalances Fixes #9972. Signed-off-by: Federico Valeri <fedevaleri@gmail.com> — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Fix TO-CC race condition issue happening during throttled rebalances This change fixes the issue caused by the TO-CC race condition happening during throttled rebalances. The TO reverts dynamic throt... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Fix TO-CC race condition issue happening during throttled rebalances This change fixes the issue caused by the TO-CC race condition happening during throttled rebalances. The TO reverts dynamic throt... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external systems,... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
- Add Topic Operator refactoring: handlers This change completes the TO refactoring by introducing a set of handlers and related tests. Each handler contains the logic used to call an external system, ... — committed to fvaleri/strimzi-kafka-operator by fvaleri 2 years ago
Hi @fvaleri, I attached the MR with the fix. I’m currently working on getting the docker build to work for me to test it locally
Yes, but only if CC integration is enabled.
https://github.com/strimzi/strimzi-kafka-operator/blob/main/topic-operator/src/main/java/io/strimzi/operator/topic/TopicOperatorConfig.java#L137