kiali: Can not fetching the proxy-status

General remarks

I configured a simple stack istio-operator, kiali-operator and prometheus (helm-chart) on the same namespace (istio-system). The issue occurs when navigating to ‘istio config’ tab in Kiali dashboard.

A brief of logs in deploy/kiali

2021-05-29T19:12:53Z WRN GetPodProxyStatus is failing for [namespace: default] [pod: details-v1-79f774bdb9-p85g4]: Error fetching the proxy-status in the following pods: istiod-77c5f5cd5-lrxg6: the server has asked for the client to provide credentials (get pods istiod-77c5f5cd5-lrxg6) 
2021-05-29T19:12:53Z WRN GetPodProxyStatus is failing for [namespace: default] [pod: productpage-v1-6b746f74dc-95zcr]: Error fetching the proxy-status in the following pods: istiod-77c5f5cd5-lrxg6: the server has asked for the client to provide credentials (get pods istiod-77c5f5cd5-lrxg6) 

Describe the bug To Reproduce

• Step1: Start a cluster as described below (all were deployed in istio-system namespace)
• Step2: Apply “getting started” of istio tutorial (Namespace: default)

kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

• Step3: Navigation to kiali-dashboard -> istio config -> can’t load istio component’s config Expected behavior
• The result should be the same as when disabling cache

  kubernetes_config:
    cache_enabled: false

a note when try to fix

• I have tried the following ways for debugging
  • #3494 -> config Security Group to open all port from anywhere
  • istioctl proxy-status

    NAME                                                   CDS        LDS        EDS        RDS        ISTIOD                     VERSION
    details-v1-79f774bdb9-p85g4.default                    SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    istio-ingressgateway-5cdfcb4956-x9ztd.istio-system     SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    productpage-v1-6b746f74dc-95zcr.default                SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    prometheus-server-648894fdf8-xfgb7.istio-system        SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    ratings-v1-b6994bb9-nq7r9.default                      SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    reviews-v1-545db77b95-lcg77.default                    SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    reviews-v2-7bf8c9648f-h4v8l.default                    SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    reviews-v3-84779c7bbc-p5fjb.default                    SYNCED     SYNCED     SYNCED     SYNCED     istiod-77c5f5cd5-lrxg6     1.11-alpha.dc12d561f6a4c97606fc4704059f5b9bcb02d9db
    

  • #3615 -> PASS when check istiod pod’s

    kubectl port-forward -n istio-system ${ISTIOD_POD_NAME} 18080:8080
    curl http://localhost:18080/debug/syncz
    

Versions used Kiali:

• Kiali operator
• Kiali CR

apiVersion: kiali.io/v1alpha1
kind: Kiali
metadata:
  name: kiali
  annotations:
    ansible.sdk.operatorframework.io/verbosity: "1"
spec:
  istio_namespace: "istio-system"

  istio_component_namespaces:
    prometheus: istio-system

  auth:
    strategy: "anonymous"

  deployment:
    accessible_namespaces: "**"
    service_type: "ClusterIP"
    pod_annotations:
      sidecar.istio.io/inject: "false"
      prometheus.io/scrape: "true"
      prometheus.io/port: "9090"
      kiali.io/runtimes: go,kiali
    hpa:
      api_version: autoscaling/v2beta2
      spec: {}
    image_pull_policy: Always
    image_pull_secrets: []
    ingress_enabled: false

  server:
    metrics_enabled: true
    metrics_port: 9090
    port: 20001
    web_root: /kiali

  external_services:
    prometheus:
      url: "http://prometheus-server.istio-system"
    custom_dashboards:
      enabled: true

Istio:

• Istio operator
• IstioOperator

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system #Istio wacher namespace
  name: overlays-profile
spec:
  profile: default
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
          service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
          proxy.istio.io/config: '{"gatewayTopology" : { "numTrustedProxies": 2 } }'
    pilot:
      k8s:
        overlays:
        - apiVersion: apps/v1
          kind: Deployment
          name: istiod
          patches:
          - path: spec.template.spec.hostNetwork
            value: true

Prometheus:

• Using helm chart without override value

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add kube-state-metrics https://kubernetes.github.io/kube-state-metrics
helm repo update
helm install prometheus prometheus-community/prometheus -n istio-system

Kubernetes flavour and version:

• AWS EKS v1.20
• Calico CNI (overlays network)