istio: Egress logs break when enabling filtering with new Telemetry API

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

We are on Istio 1.17.8 and trying to filter out healthcheck logs from AWS Route53 healthchecks. According to the docs, I’ve setup a new Telemetry object using the below YAML:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-defaults
  namespace: istio-config
spec:
  accessLogging:
  - filter:
      expression: '!(request.useragent.startsWith("Amazon-Route53-Health-Check-Service"))'
    match:
      mode: CLIENT
    providers:
      - name: envoy

And it does work…however, egress logs break for some reason. I get the below log when doing a curl https://www.google.com from my Istio-injected pod:

[2023-10-23T21:14:44.179Z] “- - -” 0 - “-” 883 24293 139 - “-” “-” “-” “-” “142.250.190.36:443” outbound|443||www.google.com 172.18.36.204:41698 240.240.100.242:443 172.18.36.204:38594 - - -

Once I apply that Telemetry object, this stops working. I’ve tried it without the match.mode block as well, same results.

Using docs here - https://istio.io/latest/docs/reference/config/telemetry/#WorkloadMode

Version

istioctl version
client version: 1.19.3
istiod version: 1.17.8
istiod version: 1.17.8
istiod version: 1.17.8

kubectl version --short
Client Version: v1.25.12
Kustomize Version: v4.5.7
Server Version: v1.25.12-eks-2d98532

Additional Information

No response