istio: telemetry: adding custom tags generates duplicated metrics

Bug Description

When adding custom tags to Istio standard metrics I get duplicated metrics.

Telemetry config:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: custom-tags
  namespace: external-istiod
spec:
  metrics:
  - overrides:
    - match:
        metric: ALL_METRICS
        mode: CLIENT
      tagOverrides:
        destination_x:
          value: upstream_peer.labels['x'].value
        source_x:
          value: labels['x'].value
    - match:
        metric: ALL_METRICS
        mode: SERVER
      tagOverrides:
        destination_x:
          value: upstream_peer.labels['x'].value
        source_x:
          value: downstream_peer.labels['x'].value
    providers:
    - name: prometheus

The output of /stats/prometheus :

istio_requests_total{response_code="500",reporter="destination",source_workload="SRC_WORKLOAD",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/default",source_app="SRC_APP",source_version="SRC_VERSION",source_cluster="SRC_CLUSTER",destination_workload="DST_WORKLOAD",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="DST_APP",destination_version="DST_VERSION",destination_service="DST_SERVICE",destination_service_name="DST_SERVICE_NAME",destination_service_namespace="default",destination_cluster="DST_CLUSTER",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="SRC_CANONICAL_SERVICE",destination_canonical_service="DST_CANONICAL_SERVICE",source_canonical_revision="SRC_CANONICAL_REVISION",destination_canonical_revision="DST_CANONICAL_REVISION",destination_x="X",source_x="X"} 480
istio_requests_total{response_code="500",reporter="destination",source_workload="SRC_WORKLOAD",source_workload_namespace="default",source_principal="spiffe://cluster.local/ns/default/sa/default",source_app="SRC_APP",source_version="SRC_VERSION",source_cluster="SRC_CLUSTER",destination_workload="DST_WORKLOAD",destination_workload_namespace="default",destination_principal="spiffe://cluster.local/ns/default/sa/default",destination_app="DST_APP",destination_version="DST_VERSION",destination_service="DST_SERVICE",destination_service_name="DST_SERVICE_NAME",destination_service_namespace="default",destination_cluster="DST_CLUSTER",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="mutual_tls",source_canonical_service="SRC_CANONICAL_SERVICE",destination_canonical_service="DST_CANONICAL_SERVICE",source_canonical_revision="SRC_CANONICAL_REVISION",destination_canonical_revision="DST_CANONICAL_REVISION"} 480

As you can see, there is two separate metrics, with the exact same value, one has the custom tags and the other does not.

Is this intended behaviour? I was expecting to have only one metric with the custom tags.

Version

$ istioctl version
client version: 1.14.1
control plane version: 1.14.1
data plane version: 1.14.1
$ kubectl version --short
Client Version: v1.22
Server Version: v1.22

Additional Information

No response

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 6
  • Comments: 16 (9 by maintainers)

Most upvoted comments

@zirain Telemetry API is fundamentally incompatible with the telemetry-focused EnvoyFilters that get applied. If one wants to use Telemetry API, the first step is really to delete those Filters. IIRC, this used to be documented somewhere, but I can’t find it now. I think for the transition period, we need to handle mapping the default providers appropriately based on flags – and properly handle upgrades with existing filters.

I’m thinking do something to avoid this, most of users will not read docs before getting errors. One of my thought is somehting like operation: INSERT_BEFORE_IF_NOEXIST, that will not apply istio.stats filter when there already have one with same named.

will review again after Telemetry API promoted to beta.