istio: Istio proxy sidecar startup probe fails when ClusterIP address/port and address/port in service entry with multiple addresses overlap

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

After upgrading from 1.21.5 to 1.22.4, istio-proxy sidecar containers were failing startup probe. Upon further investigation, the isito-proxy containers were healthy when upgraded to 1.22.3. The issue was caused by a service entry with multiple addresses and one of those addresses matched the ClusterIP of an existing service, with both of them also having the same port.

Version

istioctl version
client version: 1.22.4
control plane version: 1.22.4
data plane version: 1.22.4 (2 proxies)

kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

Additional Information

  1. In a kind cluster installed with Istio 1.22.3 default profile and default namespace labelled with istio-injection: enabled, create an nginx deployment and expose it on port 443
kubectl create deploy nginx --image=nginx
deployment.apps/nginx created

kubectl expose deploy nginx --port=443   
service/nginx exposed
  1. Create a service entry with two addresses, one with a random address, the other the ClusterIP of the nginx service (for my case, the random address is 100.110.227.28 and the nginx address is 10.96.82.33 on port 443
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: test-se
spec:
  addresses:
    - 100.110.227.28
    - 10.96.82.33
  hosts:
    - test
  ports:
    - name: TCP-443
      number: 443
      protocol: TCP
EOF
serviceentry.networking.istio.io/test-se created
  1. Rollout restart the nginx deployment. istio-proxy container should be running properly after restart
kubectl rollout restart deploy
deployment.apps/nginx restarted

kubectl get po
NAME                     READY   STATUS    RESTARTS   AGE
nginx-84949456fb-z2h5m   2/2     Running   0          63s
  1. Upgrade Istio to 1.22.4. Once the istio-ingressgateway and istiod pods have restarted, rollout restart the nginx deployment. istio-proxy container startup probe will fail
kubectl rollout restart deploy
deployment.apps/nginx restarted

kubectl get po
NAME                     READY   STATUS    RESTARTS   AGE
nginx-84949456fb-z2h5m   2/2     Running   0          2m48s
nginx-d965c6b48-dkn5r    1/2     Running   0          14s

kubectl describe po nginx-d965c6b48-dkn5r
...
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  61s                 default-scheduler  Successfully assigned default/nginx-d965c6b48-dkn5r to kind-control-plane
  Normal   Pulled     60s                 kubelet            Container image "docker.io/istio/proxyv2:1.22.4" already present on machine
  Normal   Created    60s                 kubelet            Created container istio-init
  Normal   Started    60s                 kubelet            Started container istio-init
  Normal   Pulling    60s                 kubelet            Pulling image "nginx"
  Normal   Pulled     58s                 kubelet            Successfully pulled image "nginx" in 1.896s (1.896s including waiting). Image size: 67690150 bytes.
  Normal   Created    58s                 kubelet            Created container nginx
  Normal   Started    58s                 kubelet            Started container nginx
  Normal   Pulled     58s                 kubelet            Container image "docker.io/istio/proxyv2:1.22.4" already present on machine
  Normal   Created    58s                 kubelet            Created container istio-proxy
  Normal   Started    57s                 kubelet            Started container istio-proxy
  Warning  Unhealthy  42s (x15 over 56s)  kubelet            Startup probe failed: HTTP probe failed with statuscode: 503

Checking istiod logs, entries like this can be seen warn delta ADS:LDS: ACK ERROR nginx-d965c6b48-dkn5r.default-4 Internal:Error adding/updating listener(s) 100.110.227.28_443: error adding listener: '100.110.227.28_443' has duplicate address '100.110.227.28:443,10.96.82.33:443' as existing listener

If the order of the addresses in the service entry were reversed, the issue is gone, but there is no listener for 100.110.227.28:443

kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: test-se
spec:
  addresses:
    - 10.96.82.33
    - 100.110.227.28
  hosts:
    - test
  ports:
    - name: TCP-443
      number: 443
      protocol: TCP
EOF
serviceentry.networking.istio.io/test-se configured

kubectl get po
NAME                    READY   STATUS    RESTARTS   AGE
nginx-d965c6b48-dkn5r   2/2     Running   0          7m41s

istioctl pc l deploy/nginx --port 443             
ADDRESSES    PORT MATCH                                DESTINATION
10.96.0.1    443  ALL                                  Cluster: outbound|443||kubernetes.default.svc.cluster.local
10.96.106.87 443  ALL                                  Cluster: outbound|443||istiod.istio-system.svc.cluster.local
10.96.43.110 443  ALL                                  Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
10.96.82.33  443  Trans: raw_buffer; App: http/1.1,h2c Route: nginx.default.svc.cluster.local:443
10.96.82.33  443  ALL                                  Cluster: outbound|443||nginx.default.svc.cluster.local

I suspect this bug was caused by this fix, where internal service instances were previously created for each [host, address] pair, but now created for each [host, address list] pair.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 18 (7 by maintainers)

Most upvoted comments

In a few days

On Mon, Jan 27, 2025 at 12:19 AM Sri @.***> wrote:

@howardjohn https://github.com/howardjohn Facing the same issue, any date for the next release?

— Reply to this email directly, view it on GitHub https://github.com/istio/istio/issues/52847#issuecomment-2612424049, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEYGXIPISQCH3U6WHNO6OT2MXTY3AVCNFSM6AAAAABNDHVVHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJSGQZDIMBUHE . You are receiving this because you were mentioned.Message ID: @.***>

We need something like this https://github.com/istio/istio/pull/52862 to not key on only the ‘primary’ IP. I think that Pr needs a few tweaks (and testing) though, just a rough idea

No currently set date but it would likely be weeks not months.

@yelinaung I am pretty sure this is working fine on latest 1.22 and 1.23 and is only broken on master and 1.24. https://github.com/istio/istio/pull/54207 should fix those

@thorweijie I can confirm this was not fixed as you said. I will take a look into this next week