istio: Istio proxy sidecar startup probe fails when ClusterIP address/port and address/port in service entry with multiple addresses overlap
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
After upgrading from 1.21.5 to 1.22.4, istio-proxy sidecar containers were failing startup probe. Upon further investigation, the isito-proxy containers were healthy when upgraded to 1.22.3. The issue was caused by a service entry with multiple addresses and one of those addresses matched the ClusterIP of an existing service, with both of them also having the same port.
Version
istioctl version
client version: 1.22.4
control plane version: 1.22.4
data plane version: 1.22.4 (2 proxies)
kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0
Additional Information
- In a kind cluster installed with Istio 1.22.3 default profile and default namespace labelled with
istio-injection: enabled, create an nginx deployment and expose it on port 443
kubectl create deploy nginx --image=nginx
deployment.apps/nginx created
kubectl expose deploy nginx --port=443
service/nginx exposed
- Create a service entry with two addresses, one with a random address, the other the ClusterIP of the nginx service (for my case, the random address is 100.110.227.28 and the nginx address is 10.96.82.33 on port 443
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: test-se
spec:
addresses:
- 100.110.227.28
- 10.96.82.33
hosts:
- test
ports:
- name: TCP-443
number: 443
protocol: TCP
EOF
serviceentry.networking.istio.io/test-se created
- Rollout restart the nginx deployment. istio-proxy container should be running properly after restart
kubectl rollout restart deploy
deployment.apps/nginx restarted
kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-84949456fb-z2h5m 2/2 Running 0 63s
- Upgrade Istio to 1.22.4. Once the istio-ingressgateway and istiod pods have restarted, rollout restart the nginx deployment. istio-proxy container startup probe will fail
kubectl rollout restart deploy
deployment.apps/nginx restarted
kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-84949456fb-z2h5m 2/2 Running 0 2m48s
nginx-d965c6b48-dkn5r 1/2 Running 0 14s
kubectl describe po nginx-d965c6b48-dkn5r
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 61s default-scheduler Successfully assigned default/nginx-d965c6b48-dkn5r to kind-control-plane
Normal Pulled 60s kubelet Container image "docker.io/istio/proxyv2:1.22.4" already present on machine
Normal Created 60s kubelet Created container istio-init
Normal Started 60s kubelet Started container istio-init
Normal Pulling 60s kubelet Pulling image "nginx"
Normal Pulled 58s kubelet Successfully pulled image "nginx" in 1.896s (1.896s including waiting). Image size: 67690150 bytes.
Normal Created 58s kubelet Created container nginx
Normal Started 58s kubelet Started container nginx
Normal Pulled 58s kubelet Container image "docker.io/istio/proxyv2:1.22.4" already present on machine
Normal Created 58s kubelet Created container istio-proxy
Normal Started 57s kubelet Started container istio-proxy
Warning Unhealthy 42s (x15 over 56s) kubelet Startup probe failed: HTTP probe failed with statuscode: 503
Checking istiod logs, entries like this can be seen
warn delta ADS:LDS: ACK ERROR nginx-d965c6b48-dkn5r.default-4 Internal:Error adding/updating listener(s) 100.110.227.28_443: error adding listener: '100.110.227.28_443' has duplicate address '100.110.227.28:443,10.96.82.33:443' as existing listener
If the order of the addresses in the service entry were reversed, the issue is gone, but there is no listener for 100.110.227.28:443
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: test-se
spec:
addresses:
- 10.96.82.33
- 100.110.227.28
hosts:
- test
ports:
- name: TCP-443
number: 443
protocol: TCP
EOF
serviceentry.networking.istio.io/test-se configured
kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-d965c6b48-dkn5r 2/2 Running 0 7m41s
istioctl pc l deploy/nginx --port 443
ADDRESSES PORT MATCH DESTINATION
10.96.0.1 443 ALL Cluster: outbound|443||kubernetes.default.svc.cluster.local
10.96.106.87 443 ALL Cluster: outbound|443||istiod.istio-system.svc.cluster.local
10.96.43.110 443 ALL Cluster: outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
10.96.82.33 443 Trans: raw_buffer; App: http/1.1,h2c Route: nginx.default.svc.cluster.local:443
10.96.82.33 443 ALL Cluster: outbound|443||nginx.default.svc.cluster.local
I suspect this bug was caused by this fix, where internal service instances were previously created for each [host, address] pair, but now created for each [host, address list] pair.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 6
- Comments: 18 (7 by maintainers)
In a few days
On Mon, Jan 27, 2025 at 12:19 AM Sri @.***> wrote:
We need something like this https://github.com/istio/istio/pull/52862 to not key on only the ‘primary’ IP. I think that Pr needs a few tweaks (and testing) though, just a rough idea
No currently set date but it would likely be weeks not months.
@yelinaung I am pretty sure this is working fine on latest 1.22 and 1.23 and is only broken on master and 1.24. https://github.com/istio/istio/pull/54207 should fix those
@thorweijie I can confirm this was not fixed as you said. I will take a look into this next week