kiali: Kiali is not showing traffic for any of the services
Describe the bug I am unable to see any traffic in the Kiali UI, hence no graphs etc. Also, I have had to apply a configmap yaml for kiali after the first deployment through the install config in Istio. Following files will explain better:
My install config: (here prometheus-k8s is our instance that we want to use, not the prometheus.istio-system.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: example-istiocontrolplane
spec:
profile: default
components:
ingressGateways:
- enabled: false
telemetry:
enabled: true
addonComponents:
kiali:
enabled: true
grafana:
enabled: false
prometheus:
enabled: false
tracing:
enabled: false
values:
kiali:
prometheusAddr: http://prometheus-k8s.monitoring.svc.cluster.local:9090
values:
global:
tracer:
zipkin:
address: istio-jaeger-collector.istio-system:9411
Oddly for me applying this doesnt reflect the prometheus url in configmap created, it still points to the prometheus.istio-system, so then I go on to manually apply the updated configmap and I restart the kiali pod:
apiVersion: v1
kind: ConfigMap
metadata:
name: kiali
namespace: istio-system
labels:
app: kiali
release: istio
data:
config.yaml: |
istio_component_namespaces:
grafana: monitoring
tracing: istio-system
istiod: istio-system
prometheus: monitoring
istio_namespace: istio-system
auth:
strategy: login
deployment:
accessible_namespaces: ['**']
login_token:
signing_key: "gEwUmcJZ1J"
server:
port: 20001
web_root: /kiali
external_services:
istio:
url_service_version: http://istiod.istio-system:15014/version
tracing:
url:
in_cluster_url: 'http://istio-jaeger-query.istio-system.svc.cluster.local:16686'
grafana:
url: http://grafana.monitoring.svc.cluster.local:3000
in_cluster_url:
prometheus:
url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
I created a service account that would put the metrics into my prometheus instance as:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
generation: 1
labels:
app: mixer
name: telemetry
namespace: istio-system
spec:
endpoints:
- port: http-monitoring
selector:
# the below should be the labels from kind: Service for the app
matchLabels:
app: mixer
To observe the traffic, I have deployed both bookinfo app and this: https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#set-up-the-cluster (in foo and bar namespaces only)
and then I curl from inside the sleep pod of ns bar to httpbin pod of ns foo as:
for i in seq 1 100; do curl httpbin.foo.svc.cluster.local:8000; done
I get no traffic at all. I am able to trace the traffic in jaeger successfully but nothing turns up in the kiali UI.
Expected behavior there should be some traffic and graphs on the Kiali UI as mentioned in the documentation.
Istio version: v1.6.3
Please let me know if you would want more config files.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 29 (6 by maintainers)
@yudiz-Manushi please open a new discussion and add some information related with your environment (kiali version, istio version, crd config…) it seems that your problem is different. Can you query to prometheus?
I can see in the image data related with kbc so it seems something related with metrixs
@Shashankft9 Thanks for the resolution summary!
So, what @douglas-reid suggested worked, basically, istio installation now does not need the telemetry component to be enabled. All the telemetry metrics are already created at the
istio-proxycontainer of mesh pods and to expose them to the prometheus dashboard, the mentionedServiceMonitorcan be used.Additionally to see the metrics beforehand, we can curl the 15090 port at path
/stats/prometheusof the mesh Pod. I can see the traffic on the Kiali dashboard now, looks good. I’ll test more and reopen the issue if necessary. Thanks all!@jmazzitelli I have opened an issue regarding the custom prometheus instance, you can also put your 2 cents if you are able to reproduce the error, here: https://github.com/istio/istio/issues/25174