istio: destination_service_namespace unknown in a lot of the istio_requests_total metric

(NOTE: This is used to report product bugs: To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities To ask questions about how to use Istio, please visit https://discuss.istio.io)

Bug description We have an Nginx proxy service accepting requests coming from the ingress gateway. The nginx service has sidecar configured. A lot of the istio_requests_total metrics reported from source contain unknown labels, such as destination_service_namespace. But some of them do report correctly.

• Query the service by destination_server. Some of the results have destination_service_name reported correctly, but some of them are unknown. Interesting point is that this is for the same service.

istio_requests_total{destination_service=~"surgery-app-server-proxy.*", reporter="source"} 

istio_requests_total{app="istio-ingressgateway",chart="gateways",connection_security_policy="unknown",destination_app="surgery-app-server-proxy",destination_canonical_revision="latest",destination_canonical_service="surgery-app-server-proxy",destination_principal="spiffe://cluster.local/ns/surgery-app-dev/sa/default",destination_service="surgery-app-server-proxy.surgery-app-dev.svc.cluster.local",destination_service_name="surgery-app-server-proxy",destination_service_namespace="surgery-app-dev",destination_version="unknown",destination_workload="surgery-app-server-proxy",destination_workload_namespace="surgery-app-dev",heritage="Tiller",instance="10.205.225.116:15090",istio="ingressgateway-quicktome-elb",job="kubernetes-pods",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-ingressgateway-quicktome-elb-866599b698-bmg2f",pod_template_hash="866599b698",quicktome="true",release="istio",reporter="source",request_protocol="http",request_url="/api/v1/gapServer/5f3820ab7f2791e7fb89faf7/polling/giveMeSomethingToDo",response_code="502",response_flags="-",service_istio_io_canonical_name="istio-ingressgateway-quicktome-elb",service_istio_io_canonical_revision="latest",source_app="istio-ingressgateway",source_canonical_revision="latest",source_canonical_service="istio-ingressgateway-quicktome-elb",source_principal="spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-quicktome-elb-service-account",source_version="unknown",source_workload="istio-ingressgateway-quicktome-elb",source_workload_namespace="istio-system"}
istio_requests_total{app="istio-ingressgateway",chart="gateways",connection_security_policy="unknown",destination_app="unknown",destination_canonical_revision="latest",destination_canonical_service="unknown",destination_principal="unknown",destination_service="surgery-app-server-proxy.surgery-app-dev.svc.cluster.local",destination_service_name="surgery-app-server-proxy.surgery-app-dev.svc.cluster.local",destination_service_namespace="unknown",destination_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",heritage="Tiller",instance="10.205.225.116:15090",istio="ingressgateway-quicktome-elb",job="kubernetes-pods",kubernetes_namespace="istio-system",kubernetes_pod_name="istio-ingressgateway-quicktome-elb-866599b698-bmg2f",pod_template_hash="866599b698",quicktome="true",release="istio",reporter="source",request_protocol="http",request_url="/api/v1/gapServer/5f3820ab7f2791e7fb89faf7/polling/giveMeSomethingToDo",response_code="503",response_flags="UH",service_istio_io_canonical_name="istio-ingressgateway-quicktome-elb",service_istio_io_canonical_revision="latest",source_app="istio-ingressgateway",source_canonical_revision="latest",source_canonical_service="istio-ingressgateway-quicktome-elb",source_principal="unknown",source_version="unknown",source_workload="istio-ingressgateway-quicktome-elb",source_workload_namespace="istio-system"}

Affected product area (please put an X in all that apply)

[ ] Docs [ ] Installation [ ] Networking [ ] Performance and Scalability [X ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure [ ] Upgrade

Expected behavior All the istio_requests_total metrics have labels populated correctly.

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version --short and helm version --short if you used Helm)

v1.7

How was Istio installed?

istioctl

Environment where the bug was observed (cloud vendor, OS, etc) EKS 1.16

Additionally, please consider running istioctl bug-report and attach the generated cluster-state tarball to this issue. Refer cluster state archive for more details.