strimzi-kafka-operator: [Bug]: TopicOperator removes throttled.replicas topic configuration of throttled reassignment/rebalance

Bug Description

When running kafka-reassign-partitions.sh with the --throttle flag or running a KafkaRebalance with replicationThrottle, Kafka will set these configurations on the topics that are currently moved: follower.replication.throttled.replicas: <currently moving replicas>, leader.replication.throttled.replicas: <currently moving replicas> (KIP-542).

The topic operator then picks up the configuration discrepancy and removes the configurations, which leads to unthrottled replica movements.

I would like to create a PR for this if you agree with the expected behavior.

Steps to reproduce

  1. Create cluster with 2 brokers, cruise control and enable DEBUG logging for topic operator
  2. Create some test KafkaTopic
  3. Create KafkaRebalance in which you remove a broker and set replicationThrottle (see Additional Context)
  4. Approve KafkaRebalance
  5. Wait until Cruise Control starts rebalance
  6. Observe in topic operator logs: DEBUG [LoopRunnable-0] BatchingTopicController:987 - Reconciliation #7(upsert) KafkaTopic(kafka/rebalance-test): Config changes [AlterConfigOp{opType=DELETE, configEntry=ConfigEntry(name=follower.replication.throttled.replicas, value=null, source=UNKNOWN, isSensitive=false, isReadOnly=false, synonyms=[], type=UNKNOWN, documentation=null)}, AlterConfigOp{opType=DELETE, configEntry=ConfigEntry(name=leader.replication.throttled.replicas, value=null, source=UNKNOWN, isSensitive=false, isReadOnly=false, synonyms=[], type=UNKNOWN, documentation=null)}]
  7. You can also check the topic configuration in Kafka to see that they are not set

Expected behavior

I guess the preferred behavior would be to ignore the topic configurations follower.replication.throttled.replicas and leader.replication.throttled.replicas if they are set on the broker, but not explicitly specified in the KafkaTopic resource spec.

If they are explicitly defined, the topic operator might be expected to overwrite them again.

Strimzi version

0.39.0, 0.40.0

Kubernetes version

v1.27.3

Installation method

Helm chart

Infrastructure

Kind, Bare-metal

Configuration files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.7.0
    replicas: 2
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.7"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator:
      logging:
        type: inline
        loggers:
          rootLogger.level: INFO
          logger.top.name: io.strimzi.operator.topic
          logger.top.level: DEBUG
          logger.toc.name: io.strimzi.operator.topic.TopicOperator
          logger.toc.level: TRACE
          logger.clients.level: DEBUG
    userOperator: {}
  cruiseControl: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: rebalance-test
  labels:
    strimzi.io/cluster: my-cluster
spec:
  topicName: rebalance-test
  partitions: 100

Additional context

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: rebalance-test
  labels:
    strimzi.io/cluster: my-cluster
spec:
  mode: remove-brokers
  brokers: [1]
  replicationThrottle: 10

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (14 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @fvaleri, I attached the MR with the fix. I’m currently working on getting the docker build to work for me to test it locally

To make the replicationThrottle from KafkaRebalance work by default, IGNORED_TOPIC_CONFIG would need to be follower.replication.throttled.replicas,leader.replication.throttled.replicas

Yes, but only if CC integration is enabled.

https://github.com/strimzi/strimzi-kafka-operator/blob/main/topic-operator/src/main/java/io/strimzi/operator/topic/TopicOperatorConfig.java#L137