istio: BlackHoleCluster HTTP metrics broken with Telemetry v2
Bug description Since switching to telemetry v2, metrics showing the BlackHoleCluster no longer work (when a service is trying to access a host not defined in its sidecar egress):
bash-5.0$ curl -v http://app.forecourt-service
* Trying 10.192.42.66:80...
* TCP_NODELAY set
* Connected to app.forecourt-service (10.192.42.66) port 80 (#0)
> GET / HTTP/1.1
> Host: app.forecourt-service
> User-Agent: curl/7.67.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 502 Bad Gateway
< location: http://app.forecourt-service/
< date: Sun, 23 Feb 2020 17:06:29 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host app.forecourt-service left intact
Metrics:
❯ curl -s http://127.0.0.1:15000/stats/prometheus | grep -i black
envoy_cluster_upstream_cx_total{cluster_name="BlackHoleCluster"} 0
envoy_cluster_upstream_rq_total{cluster_name="BlackHoleCluster"} 0
envoy_cluster_upstream_cx_active{cluster_name="BlackHoleCluster"} 0
Expected behavior The original feature to work as per https://github.com/istio/istio/issues/7669
Steps to reproduce the bug Use telemetry v2, hit a host that is not in your Sidecar egress host
Version (include the output of istioctl version --remote
and kubectl version
and helm version
if you used Helm)
1.4.5
How was Istio installed?
Environment where bug was observed (cloud vendor, OS, etc)
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 26 (26 by maintainers)
@gargnupur could you update the release notes for 1.5.1 and then close this out? https://github.com/istio/istio.io/blob/master/content/en/news/releases/1.5.x/announcing-1.5.1/index.md
Just a one liner. Thank you
ah ok I get why it worked before and it doesn’t work now, the metrics are not standard. We do not actually send the traffic to blackhole cluster but we report that stats as if we did