istio: Telemetry: An error has occurred during metrics gathering: label dimensions inconsistent with previously collected metrics in the same metric family

Describe the bug Istio Telemetry starts to fail on /metrics endpoint for prometheus metrics

Expected behavior Consistent labels should exist

Steps to reproduce the bug Install istio in a system that emits metrics and wait I guess?

Version Kubernets 1.9 + Istio 1.0

Installation Helm installation with slight modification

Environment AWS w/ Kube-AWS

Cluster state The amount of data the tool exposes about our cluster is unacceptable for me to post here.

Full error:

An error has occurred during metrics gathering:

2 error(s) occurred:
* collected metric istio_request_bytes label:<name:"connection_security_policy" value:"unknown" > label:<name:"destination_app" value:"permissions" > label:<name:"destination_principal" value:"unknown" > label:<name:"destination_service" value:"permissions-public.applications.svc.cluster.local" > label:<name:"destination_service_name" value:"permissions-public" > label:<name:"destination_service_namespace" value:"applications" > label:<name:"destination_version" value:"8" > label:<name:"destination_version" value:"8" > label:<name:"destination_workload" value:"permissions-grpc-v008" > label:<name:"reporter" value:"source" > label:<name:"request_protocol" value:"grpc" > label:<name:"response_code" value:"200" > label:<name:"source_app" value:"benefits" > label:<name:"source_principal" value:"unknown" > label:<name:"source_version" value:"unknown" > label:<name:"source_workload" value:"benefits-grpc" > label:<name:"source_workload_namespace" value:"applications" > histogram:<sample_count:105 sample_sum:4515 bucket:<cumulative_count:0 upper_bound:1 > bucket:<cumulative_count:0 upper_bound:10 > bucket:<cumulative_count:105 upper_bound:100 > bucket:<cumulative_count:105 upper_bound:1000 > bucket:<cumulative_count:105 upper_bound:10000 > bucket:<cumulative_count:105 upper_bound:100000 > bucket:<cumulative_count:105 upper_bound:1e+06 > bucket:<cumulative_count:105 upper_bound:1e+07 > bucket:<cumulative_count:105 upper_bound:1e+08 > >  has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric istio_request_bytes label:<name:"connection_security_policy" value:"unknown" > label:<name:"destination_principal" value:"unknown" > label:<name:"destination_service" value:"sessions-grpc-gateway.applications.svc.cluster.local" > label:<name:"destination_service_name" value:"sessions-grpc-gateway" > label:<name:"destination_service_namespace" value:"applications" > label:<name:"destination_version" value:"unknown" > label:<name:"destination_version" value:"unknown" > label:<name:"destination_workload" value:"sessions" > label:<name:"destination_workload_namespace" value:"applications" > label:<name:"reporter" value:"source" > label:<name:"request_protocol" value:"http" > label:<name:"response_code" value:"200" > label:<name:"source_app" value:"unknown" > label:<name:"source_principal" value:"unknown" > label:<name:"source_version" value:"unknown" > label:<name:"source_workload" value:"sessions" > label:<name:"source_workload_namespace" value:"applications" > histogram:<sample_count:29 sample_sum:275 bucket:<cumulative_count:28 upper_bound:1 > bucket:<cumulative_count:28 upper_bound:10 > bucket:<cumulative_count:28 upper_bound:100 > bucket:<cumulative_count:29 upper_bound:1000 > bucket:<cumulative_count:29 upper_bound:10000 > bucket:<cumulative_count:29 upper_bound:100000 > bucket:<cumulative_count:29 upper_bound:1e+06 > bucket:<cumulative_count:29 upper_bound:1e+07 > bucket:<cumulative_count:29 upper_bound:1e+08 > >  has label dimensions inconsistent with previously collected metrics in the same metric family