koperator: Problem with CC topic creation

Describe the bug Once in a while, when I create zookeeper and then KafkaCluster, operator can’t create CC topicm because it already exists. Zookeeper is fresh new with its own disks so KafkaCluster is. When I list topics via kafka tools, it is true, that CC topic exists. This happened like in 4 out of 30 instances I created. Frist it gives this this error log

{"level":"info","ts":1577806999.9847436,"logger":"controllers.KafkaCluster","msg":"A new resource was not found or may not be ready","kafkacluster":"riv/kafka","Request.Name":"kafka"}
{"level":"info","ts":1577806999.9847531,"logger":"controllers.KafkaCluster","msg":"not enough brokers available (at least three needed) for CC topic: admission webhook \"kafkatopics.kafka.banzaicloud.io\" denied the request: Replication factor is larger than the number of nodes in the kafka cluster","kafkacluster":"riv/kafka","Request.Name":"kafka"}
{"level":"info","ts":1577806999.9848156,"logger":"controllers.KafkaCluster","msg":"Reconciling KafkaCluster","kafkacluster":"riv/kafka","Request.Name":"kafka"}

After all brokers come live it gives

{"level":"info","ts":1577807042.9541788,"logger":"controllers.KafkaCluster","msg":"CR status updated","kafkacluster":"riv/kafka","Request.Name":"kafka","component":"kafka-cruisecontrol","status":"CruiseControlTopicNotReady"}
{"level":"info","ts":1577807042.954241,"logger":"controllers.KafkaCluster","msg":"could not create cruise control topic: admission webhook \"kafkatopics.kafka.banzaicloud.io\" denied the request: Topic '__CruiseControlMetrics' already exists on kafka cluster 'kafka'","kafkacluster":"riv/kafka","Request.Name":"kafka"}
{"level":"error","ts":1577807042.9542706,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"kafkacluster","request":"riv/kafka","error":"could not create cruise control topic: admission webhook \"kafkatopics.kafka.banzaicloud.io\" denied the request: Topic '__CruiseControlMetrics' already exists on kafka cluster 'kafka'","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}

After that I need to delete sts of zookeeper, delete kind: KafkaCluster and their pv and pvc. Then I create the same way zookeeper and kaflacluster and everything is fine.

I need to mention that CruiseControl deployment was not created. Only after steps above operator was able to create cruisecontrol and its topic via CRD.

It is really strange because on most of namespaces it worked fine. I use one operator to handle all namespaces with kafkacluster.

Steps to reproduce the issue: Create zookeeper sts Create KafkaCluster with 3 brokers

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 1
  • Comments: 20 (8 by maintainers)

Most upvoted comments

we are also having the same issue, 8 out of 10 times I am having this issue. Every time we have the issue we are forced to delete all Kafka cluster and zookeeper resources and recreate again.

Happened to me also today. Main problem was that CC couldn’t start. There was no deployment for CC. But CC topic was created in kafka after I listed all topic via kafka tools

This leads me to another question whether there is option to not to use CC in this setup.