eventing-kafka-broker: Stateful set implementation of KafkaSource not working

Describe the bug Upgrading eventing kafka components (controller, broker, source) from v1.5.8 to v1.6.9 doesn’t seem to be working. The older deployment of KafkaSource is still running while the statefull set remains at 0/0 pods. When the stateful sets pod tried starting initially it warned 'MountVolume.Setup failed for volume "contract-resources" : configmap "kafka-source-dispatcher-0" not found.

After running the post-install job the logs showed {"severity":"EMERGENCY","timestamp":"2022-10-25T19:33:19.923070308Z","logger":"kafka-broker-post-install","caller":"post-install/main.go:46","message":"source dispatcher deletion failed: failed while waiting for statefulset to come up: failed to get statefulset knative-eventing/kafka-source-dispatcher: statefulsets.apps \"kafka-source-dispatcher\" is forbidden: User \"system:serviceaccount:knative-eventing:knative-kafka-controller-post-install\" cannot get resource \"statefulsets\" in API group \"apps\" in the namespace \"knative-eventing\"","stacktrace":"main.main\n\tknative.dev/eventing-kafka-broker/control-plane/cmd/post-install/main.go:46\nruntime.main\n\truntime/proc.go:250"}

Expected behavior I wasn’t able to find much about this new implementation to know what should have happened, but I expected the KafkaSource stateful set to ‘take over’ the older KafkaSource deployment. I expected the post-install job to to remove the old kafkaSource implementation.

To Reproduce starting with v1.5.8 eventing kafka broker components (controller, broker, source), upgrade to 1.6.9 with the following commands:

kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.6.9/eventing-kafka-controller.yaml

kubectl apply -f https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.6.9/eventing-kafka-source.yaml

kubectl apply --filename https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.6.9/eventing-kafka-broker.yaml

https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.6.9/eventing-kafka-post-install.yaml

Knative release version 1.6.9 Additional context Add any other context about the problem here such as proposed priority

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 31 (21 by maintainers)

Most upvoted comments

@pierDipi Thanks! This seems to have fixed the initial problem I was having. I have a few more questions though.

1.) After applying the update, I see the following errors in the kafka-controller logs: {"level":"error","ts":"2022-11-18T16:55:44.956Z","logger":"kafka-broker-controller","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"3d507f2","knative.dev/pod":"kafka-controller-5f99c8bc78-cjgll","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.consumergroup.Reconciler","knative.dev/kind":"internal.kafka.eventing.knative.dev.ConsumerGroup","knative.dev/traceid":"b26cd645-27fe-467f-bdfe-4ab856e7a1fb","knative.dev/key":"dev/894c28fc-6800-4f75-bf10-f99ca1e19f30","duration":0.136223835,"error":"admission webhook \"validation.webhook.kafka.eventing.knative.dev\" denied the request: decoding request failed: cannot decode incoming new object: json: unknown field \"AuthSpec\"","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20221107171117-0243d641354d/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20221107171117-0243d641354d/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20221107171117-0243d641354d/controller/controller.go:491"}

2.) The kafka-source-dispatcher pods scale down to zero, when additional messages come in they have to scale back up and this is drastically increasing latency. Is this the intended behavior? Is the default minScale just set to zero? Should changing minScale in config-kafka-source-defaults change this behavior?

jtucci on Nov 18, 2022

In the meantime you can update the released YAMLs with that patch: https://github.com/knative-sandbox/eventing-kafka-broker/pull/2751/files (ClusterRole knative-kafka-controller-post-install in the eventing-kafka-post-install.yaml)

pierDipi on Oct 26, 2022