istio: BlackHoleCluster HTTP metrics broken with Telemetry v2

Bug description Since switching to telemetry v2, metrics showing the BlackHoleCluster no longer work (when a service is trying to access a host not defined in its sidecar egress):

bash-5.0$ curl -v http://app.forecourt-service
*   Trying 10.192.42.66:80...
* TCP_NODELAY set
* Connected to app.forecourt-service (10.192.42.66) port 80 (#0)
> GET / HTTP/1.1
> Host: app.forecourt-service
> User-Agent: curl/7.67.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< location: http://app.forecourt-service/
< date: Sun, 23 Feb 2020 17:06:29 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host app.forecourt-service left intact

Metrics:

❯ curl -s http://127.0.0.1:15000/stats/prometheus | grep -i black
envoy_cluster_upstream_cx_total{cluster_name="BlackHoleCluster"} 0
envoy_cluster_upstream_rq_total{cluster_name="BlackHoleCluster"} 0
envoy_cluster_upstream_cx_active{cluster_name="BlackHoleCluster"} 0

Expected behavior The original feature to work as per https://github.com/istio/istio/issues/7669

Steps to reproduce the bug Use telemetry v2, hit a host that is not in your Sidecar egress host

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm) 1.4.5

How was Istio installed?

Environment where bug was observed (cloud vendor, OS, etc)

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 26 (26 by maintainers)

Most upvoted comments

@gargnupur could you update the release notes for 1.5.1 and then close this out? https://github.com/istio/istio.io/blob/master/content/en/news/releases/1.5.x/announcing-1.5.1/index.md

Just a one liner. Thank you

ah ok I get why it worked before and it doesn’t work now, the metrics are not standard. We do not actually send the traffic to blackhole cluster but we report that stats as if we did