eventing-kafka-broker: Simple Broker does not become Ready, 502
Hi all, we have a wierd issue with a simple broker not becoming ready:
kn broker list
NAME URL AGE CONDITIONS READY REASON
onboarding-ci-kn-kafka-cluster 13m 4 OK / 7 False ProbeStatus : status: NotReady
Looking at the logs from kafka-controller we see that the Probe fails due to bad gateway 502:
kafka-controller-7bc844bb6b-x4frd controller {"level":"debug","ts":"2022-04-20T11:19:23.200Z","logger":"kafka-broker-controller","caller":"broker/broker.go:223","msg":"Updated dispatcher pod annotation","knative.dev/pod":"kafka-controller-7bc844bb6b-x4frd","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.broker.Reconciler","knative.dev/kind":"eventing.knative.dev.Broker","knative.dev/traceid":"eae1957d-361e-4364-a22b-afa2af7241a2","knative.dev/key":"knative-eventing/onboarding-ci-kn-kafka-cluster","action":"reconcile","uuid":"88c7538e-5722-4d83-88ea-bdf01191af7d"}
kafka-controller-7bc844bb6b-x4frd controller {"level":"info","ts":"2022-04-20T11:19:23.200Z","logger":"kafka-broker-controller","caller":"controller/controller.go:543","msg":"Reconcile succeeded","knative.dev/pod":"kafka-controller-7bc844bb6b-x4frd","knative.dev/controller":"knative.dev.eventing-kafka-broker.control-plane.pkg.reconciler.broker.Reconciler","knative.dev/kind":"eventing.knative.dev.Broker","knative.dev/traceid":"eae1957d-361e-4364-a22b-afa2af7241a2","knative.dev/key":"knative-eventing/onboarding-ci-kn-kafka-cluster","duration":0.337743497}
kafka-controller-7bc844bb6b-x4frd controller {"level":"debug","ts":"2022-04-20T11:19:23.200Z","logger":"kafka-broker-controller","caller":"prober/prober.go:63","msg":"Sending probe request","knative.dev/pod":"kafka-controller-7bc844bb6b-x4frd","scope":"prober","pod.metadata.name":"kafka-broker-receiver-7549f88579-jcmk6","address":"http://100.101.218.144:8080/knative-eventing/onboarding-ci-kn-kafka-cluster"}
kafka-controller-7bc844bb6b-x4frd controller {"level":"info","ts":"2022-04-20T11:19:23.201Z","logger":"kafka-broker-controller","caller":"prober/prober.go:86","msg":"Resource not ready","knative.dev/pod":"kafka-controller-7bc844bb6b-x4frd","scope":"prober","pod.metadata.name":"kafka-broker-receiver-7549f88579-jcmk6","address":"http://100.101.218.144:8080/knative-eventing/onboarding-ci-kn-kafka-cluster","statusCode":502}
So the IP is for kafka-broker-receiver pod. To be honest we have no clue what might be wrong here. Also seems wierd to probe the pod directly instead of a service
Expected behavior The broker to become ready
To Reproduce Steps to reproduce the behavior.
Knative release version 1.3.0 Additional context Add any other context about the problem here such as proposed priority
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (8 by maintainers)
I think we should go ahead and test this. Moreover, it makes sense for us to have knative-eventing within istio mesh, as the issues will kind of propagate to knative serving, assuming these
kservicesare sinks etc. and all our knative-serving resources are within istio-mesh. wdyt @sel-vcc @markhulia ?Also, thanks for the quick fix @pierDipi . We will report back to you to verify that things work as expected
Yes, it should work ok with a k8s
Service. As I understand it knative-eventing no-longer has a dependency on Istio, so there is noVirtualServiceforkafka-broker-ingress.Hi @matzew ! Sorry for the late reply. The patch worked for us, thanks!
Here’s a gist with patched artifacts https://gist.github.com/pierDipi/b584b0a9167dfeeffd0f934847c1dffa (you have to scroll a bit to find all the files you might need, probably it’s only
eventing-kafka-controller.yaml,eventing-kafka-broker.yaml)No, it’s targeting the service
kafka-broker-ingressinknative-eventingnow [1] [2] [3][1] https://github.com/knative-sandbox/eventing-kafka-broker/pull/2112/files#diff-56654645ee38a0dbe580207552e347b02dfbdb935ec08b38b135228bf023be42R94-R98 [2] https://github.com/knative-sandbox/eventing-kafka-broker/pull/2112/files#diff-de9e7aaae77d110f460820d7ffb652be8d38df410c584212fa11f551fae37db7R92 [3] https://github.com/knative-sandbox/eventing-kafka-broker/blob/bc251a50e7c7101c7ecdf76799e8c78ba9c72101/control-plane/config/eventing-kafka-broker/200-controller/500-controller.yaml#L83-L84
The unit tests are still using pods [4] because the prober library is unaware of the target hosts (whether they are IPs or names)
[4] https://github.com/knative-sandbox/eventing-kafka-broker/pull/2112/files#diff-6ac8e9f76343734944228a30bdfaaf8b32657d36d5ebc2606382ffb45b28fdbeR135-R146
Thanks @pierDipi, we can definitely test the patch.
I had a quick look through the PR and if I have understood correctly the probing is still based on the Pod IP addresses? We can certainly test the fix, but I’m fairly sure that we cannot connect to those IPs from within Istio. The reason behind this is that the Envoy config provided by Istio is based on the k8s service DNS address, which Envoy can resolve to an IP and match against an incoming request’s authority. However, Envoy does not know about the relationship between the k8s Service and the set of Pods that back it, so Envoy has no upstream config for those Pod IPs and returns a 502 response.
I’ve created a patch in https://github.com/knative-sandbox/eventing-kafka-broker/pull/2112, after CI jobs run and they are green, is anyone willing to test the patch with Istio and your setup (I will give you custom manifests unless you want to build the project from source code)?
Hi @pierDipi, In this case both
kafka-controllerandkafka-broker-receiverpods were part of the istio mesh (injected with istio-proxy sidecar containers). The issue is that it is not possible to connect to a Pod IP address because there is noVirtualServiceto define the route.I have confirmed this behaviour with istio’s sleep and httpbin samples deployed to a namespace with istio-injection enabled:-