kiali: Kiali is not showing traffic for any of the services

Describe the bug I am unable to see any traffic in the Kiali UI, hence no graphs etc. Also, I have had to apply a configmap yaml for kiali after the first deployment through the install config in Istio. Following files will explain better:

My install config: (here prometheus-k8s is our instance that we want to use, not the prometheus.istio-system.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: default
  components:
    ingressGateways:
    - enabled: false
    telemetry:
      enabled: true
  addonComponents:
    kiali:
      enabled: true
    grafana:
      enabled: false
    prometheus:
      enabled: false
    tracing:
      enabled: false
  values:
    kiali:
      prometheusAddr: http://prometheus-k8s.monitoring.svc.cluster.local:9090
  values:
    global:
      tracer:
        zipkin:
          address: istio-jaeger-collector.istio-system:9411

Oddly for me applying this doesnt reflect the prometheus url in configmap created, it still points to the prometheus.istio-system, so then I go on to manually apply the updated configmap and I restart the kiali pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kiali
  namespace: istio-system
  labels:
    app: kiali
    release: istio
data:
  config.yaml: |
    istio_component_namespaces:
      grafana: monitoring
      tracing: istio-system
      istiod: istio-system
      prometheus: monitoring
    istio_namespace: istio-system
    auth:
      strategy: login
    deployment:
      accessible_namespaces: ['**']
    login_token:
      signing_key: "gEwUmcJZ1J"
    server:
      port: 20001
      web_root: /kiali
    external_services:
      istio:
        url_service_version: http://istiod.istio-system:15014/version
      tracing:
        url:
        in_cluster_url: 'http://istio-jaeger-query.istio-system.svc.cluster.local:16686'
      grafana:
        url: http://grafana.monitoring.svc.cluster.local:3000
        in_cluster_url:
      prometheus:
        url: http://prometheus-k8s.monitoring.svc.cluster.local:9090

I created a service account that would put the metrics into my prometheus instance as:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  generation: 1
  labels:
    app: mixer
  name: telemetry
  namespace: istio-system
spec:
  endpoints:
  - port: http-monitoring
  selector:
    # the below should be the labels from kind: Service for the app
    matchLabels:
      app: mixer

To observe the traffic, I have deployed both bookinfo app and this: https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster (in foo and bar namespaces only)

and then I curl from inside the sleep pod of ns bar to httpbin pod of ns foo as: for i in seq 1 100; do curl httpbin.foo.svc.cluster.local:8000; done

I get no traffic at all. I am able to trace the traffic in jaeger successfully but nothing turns up in the kiali UI.

Expected behavior there should be some traffic and graphs on the Kiali UI as mentioned in the documentation.

Istio version: v1.6.3

Please let me know if you would want more config files.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 29 (6 by maintainers)

Most upvoted comments

@yudiz-Manushi please open a new discussion and add some information related with your environment (kiali version, istio version, crd config…) it seems that your problem is different. Can you query to prometheus?

I can see in the image data related with kbc so it seems something related with metrixs

@Shashankft9 Thanks for the resolution summary!

So, what @douglas-reid suggested worked, basically, istio installation now does not need the telemetry component to be enabled. All the telemetry metrics are already created at the istio-proxy container of mesh pods and to expose them to the prometheus dashboard, the mentioned ServiceMonitor can be used.

Additionally to see the metrics beforehand, we can curl the 15090 port at path /stats/prometheus of the mesh Pod. I can see the traffic on the Kiali dashboard now, looks good. I’ll test more and reopen the issue if necessary. Thanks all!

@jmazzitelli I have opened an issue regarding the custom prometheus instance, you can also put your 2 cents if you are able to reproduce the error, here: https://github.com/istio/istio/issues/25174