istio: host header is not being sent with probes after upgrading to 1.18.1

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

After upgrading from istio 1.16.1 to 1.18.1 my pods are no longer receiving the correct Host header on their startup/readiness/liveness probes. I have inconsistent results with other custom headers. I have a few pods that are receiving custom headers, but most of them are not receiving those as well. All with the same pod spec.

This configuration worked with istio 1.16.1. I have not tested intermediate versions.

I modified the app to ignore the host header to debug this and can confirm that the probe is hitting the correct endpoint on our app. However, we expect the host header to be sent in the request.

Readiness probe in Deployment

        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: Host
              value: example.com
            - name: X-Request-Id
              value: readiness-probe
            path: /__health
            port: 5000
            scheme: HTTP
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 1

Readiness probe in auto-injected Pod

    readinessProbe:
      failureThreshold: 3
      httpGet:
        httpHeaders:
        - name: Host
          value: example.com
        - name: X-Request-Id
          value: readiness-probe
        path: /app-health/web/readyz
        port: 15020
        scheme: HTTP
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 1

ISTIO_KUBE_APP_PROBERS

    - name: ISTIO_KUBE_APP_PROBERS
      value: '{"/app-health/web/livez":{"httpGet":{"path":"/__health","port":5000,"scheme":"HTTP","httpHeaders":[{"name":"Host","value":"example.com"},{"name":"X-Request-Id","value":"liveness-probe"}]},"timeoutSeconds":1},"/app-health/web/readyz":{"httpGet":{"path":"/__health","port":5000,"scheme":"HTTP","httpHeaders":[{"name":"Host","value":"example.com"},{"name":"X-Request-Id","value":"readiness-probe"}]},"timeoutSeconds":1},"/app-health/web/startupz":{"httpGet":{"path":"/__health","port":5000,"scheme":"HTTP","httpHeaders":[{"name":"Host","value":"example.com"},{"name":"X-Request-Id","value":"startup-probe"}]},"timeoutSeconds":1}}'

Captured tcpdump

• You can see that the Host header being sent is 10.34.54.17:5000.
• You can also see the X-Request-Id is the expected readiness-probe value.

$ tcpdump -i lo -X port 5000
<snip>
        0x0030:  f3a7 1fa5 4745 5420 2f5f 5f68 6561 6c74  ....GET./__healt
        0x0040:  6820 4854 5450 2f31 2e31 0d0a 486f 7374  h.HTTP/1.1..Host
        0x0050:  3a20 3130 2e33 342e 3534 2e31 373a 3530  :.10.34.54.17:50
        0x0060:  3030 0d0a 5573 6572 2d41 6765 6e74 3a20  00..User-Agent:.
        0x0070:  6b75 6265 2d70 726f 6265 2f31 2e32 362b  kube-probe/1.26+
        0x0080:  0d0a 4163 6365 7074 3a20 2a2f 2a0d 0a43  ..Accept:.*/*..C
        0x0090:  6f6e 6e65 6374 696f 6e3a 2063 6c6f 7365  onnection:.close
        0x00a0:  0d0a 582d 5265 7175 6573 742d 4964 3a20  ..X-Request-Id:.
        0x00b0:  7265 6164 696e 6573 732d 7072 6f62 650d  readiness-probe.
        0x00c0:  0a41 6363 6570 742d 456e 636f 6469 6e67  .Accept-Encoding
        0x00d0:  3a20 677a 6970 0d0a 0d0a                 :.gzip....

Version

$ istioctl version
client version: 1.17.3
control plane version: 1.18.1
data plane version: 1.16.1 (1 proxies), 1.18.1 (47 proxies)

Note that the 1 proxy on 1.16.1 is the old version of the pod I'm trying to redeploy with 1.18.1 but the startup probe is failing due to this issue.

$ kubectl version --short
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.26.6-eks-a5565ad

Additional Information

Probably related: https://github.com/istio/istio/pull/45632 and https://github.com/istio/istio/issues/45482

I should probably emphasize again that I have only 1 pod where the x-request-id custom headers are working out of 4 running examples (across 2 separate kubernetes clusters). And every pod has a broken host header.

Affected product area

• Ambient
• Docs
• Installation
• Networking
• Performance and Scalability
• Extensions and Telemetry
• Security
• Test and Release
• User Experience
• Developer Infrastructure
• Upgrade
• Multi Cluster
• Virtual Machine
• Control Plane Revisions