istio: telemetry: adding custom tags generates duplicated metrics
Bug Description
When adding custom tags to Istio standard metrics I get duplicated metrics.
Telemetry config:
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
name: custom-tags
namespace: external-istiod
spec:
metrics:
- overrides:
- match:
metric: ALL_METRICS
mode: CLIENT
tagOverrides:
destination_x:
value: upstream_peer.labels['x'].value
source_x:
value: labels['x'].value
- match:
metric: ALL_METRICS
mode: SERVER
tagOverrides:
destination_x:
value: upstream_peer.labels['x'].value
source_x:
value: downstream_peer.labels['x'].value
providers:
- name: prometheus
The output of /stats/prometheus :
istio_requests_total{response_code="500",reporter="destination",source_workload="SRC_WORKLOAD",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/default",source_app="SRC_APP",source_version="SRC_VERSION",source_cluster="SRC_CLUSTER",destination_workload="DST_WORKLOAD",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="DST_APP",destination_version="DST_VERSION",destination_service="DST_SERVICE",destination_service_name="DST_SERVICE_NAME",destination_service_namespace="default",destination_cluster="DST_CLUSTER",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="SRC_CANONICAL_SERVICE",destination_canonical_service="DST_CANONICAL_SERVICE",source_canonical_revision="SRC_CANONICAL_REVISION",destination_canonical_revision="DST_CANONICAL_REVISION",destination_x="X",source_x="X"} 480
istio_requests_total{response_code="500",reporter="destination",source_workload="SRC_WORKLOAD",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/default",source_app="SRC_APP",source_version="SRC_VERSION",source_cluster="SRC_CLUSTER",destination_workload="DST_WORKLOAD",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="DST_APP",destination_version="DST_VERSION",destination_service="DST_SERVICE",destination_service_name="DST_SERVICE_NAME",destination_service_namespace="default",destination_cluster="DST_CLUSTER",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="SRC_CANONICAL_SERVICE",destination_canonical_service="DST_CANONICAL_SERVICE",source_canonical_revision="SRC_CANONICAL_REVISION",destination_canonical_revision="DST_CANONICAL_REVISION"} 480
As you can see, there is two separate metrics, with the exact same value, one has the custom tags and the other does not.
Is this intended behaviour? I was expecting to have only one metric with the custom tags.
Version
$ istioctl version
client version: 1.14.1
control plane version: 1.14.1
data plane version: 1.14.1
$ kubectl version --short
Client Version: v1.22
Server Version: v1.22
Additional Information
No response
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 6
- Comments: 16 (9 by maintainers)
I’m thinking do something to avoid this, most of users will not read docs before getting errors. One of my thought is somehting like
operation: INSERT_BEFORE_IF_NOEXIST, that will not applyistio.statsfilter when there already have one with same named.will review again after Telemetry API promoted to beta.