kiali: Kiali does not work well with Google Anthos Service Mesh

Hello team,

We are on Google’s GKE Kubernetes cluster. And we are also using Google’s Istio offering called Anthos Service Mesh. This is a managed offering of istio where the control plane is managed by Google.

We deployed Kiali through the Kiali operator and CRD and we have seen that when it’s just a handful of workloads, it can show us the required information about graphs and other visualization.

However, when we added all our workloads into the mesh, Kali became unusable and we were not able to see any traffic visualization

We tried bumping the resources but that did not help either.

All we see in the kiali pods are the following warning

kiali-7c79cb774b-hc8jr kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: platform-metrics] [pod: vmstorage-jldp-14]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-4rn9h kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: mobile-apps] [pod: stubs-things-we-love-64b7d98866-945tl]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-hc8jr kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: enablement] [pod: stubs-greetings-ui-7f4fffd7f-rfvhl]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-4rn9h kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: platform-metrics] [pod: vmselect-jldp-7]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-4rn9h kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: mobile-apps] [pod: stubs-things-we-love-64b7d98866-vw82w]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-hc8jr kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: catalogue-feeds] [pod: stubs-interface-01p-collator-5969f45c7-4rw69]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-hc8jr kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: mobile-apps] [pod: stubs-skus-7dfc5b4b89-28xzz]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-prdrj kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: catalogue] [pod: stubs-ratings-collator-6f65fffd7b-nfbd4]: unable to find any healthy Pilot instance "}
kiali-7c79cb774b-4rn9h kiali {"level":"warn","time":"2022-08-23T12:09:16Z","message":"GetPodProxyStatus is failing for [namespace: platform] [pod: microservice-manager-webhooks-697c497fb-qzwvh]: unable to find any healthy Pilot instance "}

We are not sure if this is the problem with Kiali being unable to visualize with a lot of workloads in the mesh or is it just the sheer size of our cluster?

Any help would be appreciated

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 29 (11 by maintainers)

Most upvoted comments

https://github.com/istio/istio/commit/0f870bce1823c13ca57fbb45aa68fe94eea60fe9 was the istioctl PR to add it

@qfel may be able to comment more

howardjohn on Oct 6, 2022

@jshaughn I think we should move forward with supporting connecting to istiod through a remote url as a config option in addition to the port forwarding solution we currently support if you’ll move this back to backlog.

nrfox on Sep 21, 2022

A note, although our compatibility matrix (on kiali.io) doesn’t show tested support, I’m fairly sure v1.48 is compatible with Istio 1.12, as that is the combination shipped with Red Hat’s OSSM product.

jshaughn on Aug 23, 2022

For the record, here’s where the error message comes from - you can see where its trying to find an istiod to talk to: https://github.com/kiali/kiali/blob/master/kubernetes/istio.go#L69

Here is the 1.42.0 codebase: https://github.com/kiali/kiali/blob/v1.42.0/kubernetes/istio.go#L65

It is looking for an istiod pod found in the istio namespace (i.e. the one that you tell Kiali about - its the istio_namespace setting in the Kiali ConfigMap, which can be defined in the Kiali CR in spec.istio_namespace). It looks for a pod with the app label set to istiod. If there are no istiod pods that are in the running state, you will get this error.

Now, you said, “Since we are using Google’s Anthos Service Mesh, istiod is managed by them. istio-system namespace does not have any pods.” – if so, that’s going to break Kiali. There needs to be an istiod pod so Kiali can talk to it. So I suspect Anthos is configuring things in a weird way that Kiali doesn’t understand. But I have a feeling all you need to do is tell Kiali where your Istio control plane is (set the spec.istio_namespace to where all the control plane pods (including istiod) are) and it “should” work.

jmazzitelli on Aug 23, 2022