istio: Pods fail to create with istio-cni 500 "no ztunnel connection"
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
After upgrading to Istio 1.24.0, we observed some (not all) pods failing to create on certain nodes. The following event was logged from the pod:
Warning FailedCreatePodSandBox 2m23s (x3160 over 11h) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "94443ef2f45d35e7e2e6af3af98977db9d07c6ac02d4dab3e216b804f878dcd4": plugin type="istio-cni" name="istio-cni" failed (add): unable to push CNI event (status code 500): partial add error: no ztunnel connection
The istio-cni logs likewise show a 500 trying to connect to ztunnel:
2024-11-09T18:51:05.030649Z info cni-agent adding pod to the mesh ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.031134Z info cni-agent Running iptables-legacy-restore with the following input:
* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
* nat
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A ISTIO_PRERT -s 169.254.7.127 -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -d 169.254.7.127 -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.031178Z info Running command (without lock by env and nss): iptables-legacy-restore --noflush -v
2024-11-09T18:51:05.033576Z info cni-agent Running ip6tables-legacy-restore with the following input:
* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
* nat
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A ISTIO_PRERT -s fd16:9254:7127:1337:ffff:ffff:ffff:ffff -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -d fd16:9254:7127:1337:ffff:ffff:ffff:ffff -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d ::1/128 -p tcp ! --dport 15008 -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d ::1/128 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d ::1/128 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.033642Z info Running command (without lock by env and nss): ip6tables-legacy-restore --noflush -v
2024-11-09T18:51:05.036149Z error cni-agent failed to add pod to ztunnel: partial add error: no ztunnel connection
2024-11-09T18:51:05.045072Z error cni-agent pod: services/account-dashboard-importer-55f6ffc6bf-cszsw was not enrolled and is unhealthy: partial add error: no ztunnel connection
2024-11-09T18:51:05.045092Z error cni-agent failed to handle add event: partial add error: no ztunnel connection ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.045434Z error cni-plugin istio-cni cmdAdd failed to signal node Istio CNI agent: unable to push CNI event (status code 500): partial add error: no ztunnel connection pod=services/account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.045482Z error cni-plugin istio-cni cmdAdd error: unable to push CNI event (status code 500): partial add error: no ztunnel connection
2024-11-09T18:51:05.045493Z error cni-plugin istio-cni failed with: unable to push CNI event (status code 500): partial add error: no ztunnel connection
The ztunnel pod does not seem to have any errors in its log, and is handling traffic for other the pods on the node.
2024-11-09T18:51:37.224763Z info access connection complete src.addr=10.0.80.178:34772 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.81.175:15008 dst.hbone_addr=10.0.81.175:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-h9kp2" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="7ms"
2024-11-09T18:51:37.383071Z info access connection complete src.addr=10.0.80.178:34784 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.96.197:15008 dst.hbone_addr=10.0.96.197:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-nx2ln" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="8ms"
2024-11-09T18:51:37.424544Z info access connection complete src.addr=10.0.80.178:34798 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.81.175:15008 dst.hbone_addr=10.0.81.175:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-h9kp2" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="8ms"
2024-11-09T18:51:37.475135Z info access connection complete src.addr=10.0.80.178:34812 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.96.197:15008 dst.hbone_addr=10.0.96.197:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-nx2ln" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="12ms"
Rolling back to 1.23.3 seems to have resolved the problem, but required draining the affected nodes and allowing new nodes to spin up.
Version
❯ istioctl version
client version: 1.23.3
control plane version: 1.24.0
data plane version: 1.24.0 (6 proxies)
❯ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.2-eks-7f9249a
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 4
- Comments: 24 (11 by maintainers)
Commits related to this issue
- Fix #54544 (and probably #53843) Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io> — committed to bleggett/istio by bleggett 2 years ago
- Handle out-of-order ztunnel connection removals correctly. (#54565) * Fix #54544 (and probably #53843) Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io> * Relnote Signed-off-by: Benjamin ... — committed to istio/istio by bleggett 2 years ago
- Fix #54544 (and probably #53843) Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io> — committed to istio-testing/istio by bleggett 2 years ago
- [release-1.24] Handle out-of-order ztunnel connection removals correctly. (#54569) * Fix #54544 (and probably #53843) Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io> * Relnote Signed-of... — committed to istio/istio by istio-testing 2 years ago
Either 1.24.3 or 1.25.0, whichever comes first.
https://github.com/istio/istio/wiki/Istio-Release-1.25 is slated for next month.
@rwong2888 Just merged a fix (I suspect) for this in https://github.com/istio/istio/pull/54565 - I’m backporting this to 1.24 so it should be in the next point release, as well as 1.25
Having trouble replicating this consistently, and need to move on to other things for a bit - if I see it happen again I will post back here! For now, I think we can resolve the problem through a restart of
istio-cni-nodepods after rolling out the update.@jaellio no, those were without setting
privilegedto true. I can try to see if we can test withprivileged: truebut it’s a little tricky because the Helm chart doesn’t expose that as a value, and Argo will revert any manual changes I make to the daemonset when I try to apply those. 🤔@keithmattix I dialed up the logs for ztunnel and istio-cni-node to debug, but didn’t observe anything new from what I originally posted:
istio-cni-node:
ztunnel, happily handling traffic and not showing any indication of a problem:
One further observation - the issue resolved itself as soon as I restarted the
istio-cni-nodepod.@jaellio Yes, some nodes are fine while others are not. Config on the nodes is all the same. I am thinking more and more it feels like a race condition. It’s hard to replicate but after a few tries switching between 1.23.3 and 1.24.0 I can get it to happen.
Oh and to clarify, from ztunnel’s standpoint, it thinks everything is fine. Ztunnel pods are all healthy, but istio-cni logs 500s.