istio: Pods fail to create with istio-cni 500 "no ztunnel connection"

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

After upgrading to Istio 1.24.0, we observed some (not all) pods failing to create on certain nodes. The following event was logged from the pod:

Warning  FailedCreatePodSandBox  2m23s (x3160 over 11h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "94443ef2f45d35e7e2e6af3af98977db9d07c6ac02d4dab3e216b804f878dcd4": plugin type="istio-cni" name="istio-cni" failed (add): unable to push CNI event (status code 500): partial add error: no ztunnel connection

The istio-cni logs likewise show a 500 trying to connect to ztunnel:

2024-11-09T18:51:05.030649Z    info    cni-agent    adding pod to the mesh    ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.031134Z    info    cni-agent    Running iptables-legacy-restore with the following input:
* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
* nat
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A ISTIO_PRERT -s 169.254.7.127 -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -d 169.254.7.127 -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT    ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.031178Z    info    Running command (without lock by env and nss): iptables-legacy-restore --noflush -v
2024-11-09T18:51:05.033576Z    info    cni-agent    Running ip6tables-legacy-restore with the following input:
* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
* nat
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A ISTIO_PRERT -s fd16:9254:7127:1337:ffff:ffff:ffff:ffff -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -d fd16:9254:7127:1337:ffff:ffff:ffff:ffff -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d ::1/128 -p tcp ! --dport 15008 -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d ::1/128 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d ::1/128 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT    ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.033642Z    info    Running command (without lock by env and nss): ip6tables-legacy-restore --noflush -v
2024-11-09T18:51:05.036149Z    error    cni-agent    failed to add pod to ztunnel: partial add error: no ztunnel connection
2024-11-09T18:51:05.045072Z    error    cni-agent    pod: services/account-dashboard-importer-55f6ffc6bf-cszsw was not enrolled and is unhealthy: partial add error: no ztunnel connection
2024-11-09T18:51:05.045092Z    error    cni-agent    failed to handle add event: partial add error: no ztunnel connection    ns=services name=account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.045434Z    error    cni-plugin    istio-cni cmdAdd failed to signal node Istio CNI agent: unable to push CNI event (status code 500): partial add error: no ztunnel connection    pod=services/account-dashboard-importer-55f6ffc6bf-cszsw
2024-11-09T18:51:05.045482Z    error    cni-plugin    istio-cni cmdAdd error: unable to push CNI event (status code 500): partial add error: no ztunnel connection
2024-11-09T18:51:05.045493Z    error    cni-plugin    istio-cni failed with: unable to push CNI event (status code 500): partial add error: no ztunnel connection

The ztunnel pod does not seem to have any errors in its log, and is handling traffic for other the pods on the node.

2024-11-09T18:51:37.224763Z    info    access    connection complete    src.addr=10.0.80.178:34772 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.81.175:15008 dst.hbone_addr=10.0.81.175:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-h9kp2" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="7ms"
2024-11-09T18:51:37.383071Z    info    access    connection complete    src.addr=10.0.80.178:34784 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.96.197:15008 dst.hbone_addr=10.0.96.197:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-nx2ln" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="8ms"
2024-11-09T18:51:37.424544Z    info    access    connection complete    src.addr=10.0.80.178:34798 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.81.175:15008 dst.hbone_addr=10.0.81.175:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-h9kp2" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="8ms"
2024-11-09T18:51:37.475135Z    info    access    connection complete    src.addr=10.0.80.178:34812 src.workload="cc-compliance-indexer-54fd5548d9-m6k4l" src.namespace="services" src.identity="spiffe://cluster.local/ns/services/sa/cc-compliance-indexer" dst.addr=10.0.96.197:15008 dst.hbone_addr=10.0.96.197:8080 dst.service="cc-subdomain-api.services.svc.cluster.local" dst.workload="cc-subdomain-api-6799c54f85-nx2ln" dst.namespace="services" dst.identity="spiffe://cluster.local/ns/services/sa/cc-subdomain-api" direction="outbound" bytes_sent=493 bytes_recv=310 duration="12ms"

Rolling back to 1.23.3 seems to have resolved the problem, but required draining the affected nodes and allowing new nodes to spin up.

Version

❯ istioctl version
client version: 1.23.3
control plane version: 1.24.0
data plane version: 1.24.0 (6 proxies)

❯ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.2-eks-7f9249a

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 24 (11 by maintainers)

Commits related to this issue

Most upvoted comments

Either 1.24.3 or 1.25.0, whichever comes first.

https://github.com/istio/istio/wiki/Istio-Release-1.25 is slated for next month.

@rwong2888 Just merged a fix (I suspect) for this in https://github.com/istio/istio/pull/54565 - I’m backporting this to 1.24 so it should be in the next point release, as well as 1.25

Having trouble replicating this consistently, and need to move on to other things for a bit - if I see it happen again I will post back here! For now, I think we can resolve the problem through a restart of istio-cni-node pods after rolling out the update.

@jaellio no, those were without setting privileged to true. I can try to see if we can test with privileged: true but it’s a little tricky because the Helm chart doesn’t expose that as a value, and Argo will revert any manual changes I make to the daemonset when I try to apply those. 🤔

@keithmattix I dialed up the logs for ztunnel and istio-cni-node to debug, but didn’t observe anything new from what I originally posted:

istio-cni-node:

2024-11-11T19:05:41.843416Z    error    cni-agent    failed to add pod to ztunnel: partial add error: no ztunnel connection
2024-11-11T19:05:41.853528Z    error    cni-agent    pod: unruly-bacon/portfolios-web-5dbc6d67f-dxqn6 was not enrolled and is unhealthy: partial add error: no ztunnel connection
2024-11-11T19:05:41.853560Z    error    cni-agent    failed to handle add event: partial add error: no ztunnel connection    ns=unruly-bacon name=portfolios-web-5dbc6d67f-dxqn6
2024-11-11T19:05:41.853821Z    error    cni-plugin    istio-cni cmdAdd failed to signal node Istio CNI agent: unable to push CNI event (status code 500): partial add error: no ztunnel connection    pod=unruly-bacon/portfolios-web-5dbc6d67f-dxqn6
2024-11-11T19:05:41.853890Z    error    cni-plugin    istio-cni cmdAdd error: unable to push CNI event (status code 500): partial add error: no ztunnel connection
2024-11-11T19:05:41.853908Z    error    cni-plugin    istio-cni failed with: unable to push CNI event (status code 500): partial add error: no ztunnel connection

ztunnel, happily handling traffic and not showing any indication of a problem:

2024-11-11T19:03:10.568082Z    info    access    connection complete    src.addr=100.100.3.8:49846 src.workload="portfolios-graphql-b7f445bbf-vl7jf" src.namespace="light-grass" src.identity="spiffe://cluster.local/ns/light-grass/sa/portfolios-graphql" dst.addr=100.100.14.218:15008 dst.hbone_addr=100.100.14.218:8080 dst.service="authentication-service.light-grass.svc.cluster.local" dst.workload="authentication-service-79d5d9694c-w7vj5" dst.namespace="light-grass" dst.identity="spiffe://cluster.local/ns/light-grass/sa/authentication-service" direction="inbound" bytes_sent=462 bytes_recv=61 duration="4014ms"
2024-11-11T19:03:10.568117Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    send    frame=Data { stream_id: StreamId(13), flags: (0x1: END_STREAM) }
2024-11-11T19:03:12.506674Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    send    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:12.507024Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    received    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:12.507040Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    send    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:12.507046Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    received    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:16.550877Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    received    frame=Headers { stream_id: StreamId(15), flags: (0x4: END_HEADERS) }
2024-11-11T19:03:16.550970Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:inbound{id=7288bbd76388f3a5d58a6af39d4b73e4 peer=100.100.3.8:49846}    wait for workload    wl=authentication-service.light-grass (authentication-service-79d5d9694c-w7vj5)
2024-11-11T19:03:16.550981Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:inbound{id=7288bbd76388f3a5d58a6af39d4b73e4 peer=100.100.3.8:49846}    got sub, waiting for workload    wl=authentication-service.light-grass (authentication-service-79d5d9694c-w7vj5)
2024-11-11T19:03:16.551001Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:inbound{id=7288bbd76388f3a5d58a6af39d4b73e4 peer=100.100.3.8:49846}    fetch workload    addr=/100.100.3.8
2024-11-11T19:03:16.551021Z    debug    access    connection opened    src.addr=100.100.3.8:49846 src.workload="portfolios-graphql-b7f445bbf-vl7jf" src.namespace="light-grass" src.identity="spiffe://cluster.local/ns/light-grass/sa/portfolios-graphql" dst.addr=100.100.14.218:15008 dst.hbone_addr=100.100.14.218:8080 dst.service="authentication-service.light-grass.svc.cluster.local" dst.workload="authentication-service-79d5d9694c-w7vj5" dst.namespace="light-grass" dst.identity="spiffe://cluster.local/ns/light-grass/sa/authentication-service" direction="inbound"
2024-11-11T19:03:16.551040Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:inbound{id=7288bbd76388f3a5d58a6af39d4b73e4 peer=100.100.3.8:49846}    no allow policies, allow    
2024-11-11T19:03:16.551264Z    debug    proxy::inbound:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:inbound{id=7288bbd76388f3a5d58a6af39d4b73e4 peer=100.100.3.8:49846}    connected to: 100.100.14.218:8080    
2024-11-11T19:03:16.551296Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    send    frame=Headers { stream_id: StreamId(15), flags: (0x4: END_HEADERS) }
2024-11-11T19:03:16.552221Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    received    frame=Data { stream_id: StreamId(15) }
2024-11-11T19:03:16.560876Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=da56dbfda709e42de6c7d34382ed6a61}:Connection{peer=Client}    send    frame=Data { stream_id: StreamId(1) }
2024-11-11T19:03:16.561009Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-redis-master-0}:Connection{peer=Server}    received    frame=Data { stream_id: StreamId(1) }
2024-11-11T19:03:16.561200Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-redis-master-0}:Connection{peer=Server}    send    frame=Data { stream_id: StreamId(1) }
2024-11-11T19:03:16.561261Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=da56dbfda709e42de6c7d34382ed6a61}:Connection{peer=Client}    received    frame=Data { stream_id: StreamId(1) }
2024-11-11T19:03:16.562048Z    debug    proxy::outbound:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    connection started    component="outbound"
2024-11-11T19:03:16.562068Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    wait for workload    wl=authentication-service.light-grass (authentication-service-79d5d9694c-w7vj5)
2024-11-11T19:03:16.562073Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    got sub, waiting for workload    wl=authentication-service.light-grass (authentication-service-79d5d9694c-w7vj5)
2024-11-11T19:03:16.562081Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    fetch address    network_addr.address=172.20.12.183
2024-11-11T19:03:16.562086Z    debug    state:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    fetch address    network_addr.address=172.20.12.183
2024-11-11T19:03:16.562095Z    debug    proxy::outbound:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    built request to workload    
2024-11-11T19:03:16.562109Z    debug    access    connection opened    src.addr=100.100.14.218:44567 src.workload="authentication-service-79d5d9694c-w7vj5" src.namespace="light-grass" src.identity="spiffe://cluster.local/ns/light-grass/sa/authentication-service" dst.addr=100.100.8.138:15008 dst.hbone_addr=100.100.8.138:3306 dst.service="authentication-service-mysql.light-grass.svc.cluster.local" dst.workload="authentication-service-mysql-0" dst.namespace="light-grass" dst.identity="spiffe://cluster.local/ns/light-grass/sa/authentication-service-mysql" direction="outbound"
2024-11-11T19:03:16.562142Z    debug    proxy::pool:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    checkout - found mutex for pool key ConnectionMeta { key: 12960813990266072314, id: 18 }, waiting for writelock    
2024-11-11T19:03:16.562151Z    debug    proxy::pool:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    re-using connection for 100.100.14.218(spiffe://cluster.local/ns/light-grass/sa/authentication-service)->100.100.8.138:15008[spiffe://cluster.local/ns/light-grass/sa/authentication-service-mysql]    
2024-11-11T19:03:16.562168Z    debug    proxy::pool:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    initial attempt - found existing conn, done    
2024-11-11T19:03:16.562189Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    send    frame=Headers { stream_id: StreamId(27), flags: (0x4: END_HEADERS) }
2024-11-11T19:03:16.562254Z    debug    pingora_pool::connection:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=6a5230455b04c6f8a94822128357fb09}    idle connection is being picked up    
2024-11-11T19:03:16.562258Z    debug    proxy::pool:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=6a5230455b04c6f8a94822128357fb09}    connection ConnectionMeta { key: 12960813990266072314, id: 17 } was removed/checked out/timed out of the pool    
2024-11-11T19:03:16.562265Z    debug    proxy::pool:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    starting an idle timeout for connection ConnectionMeta { key: 12960813990266072314, id: 18 }    
2024-11-11T19:03:16.562591Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    received    frame=Headers { stream_id: StreamId(27), flags: (0x4: END_HEADERS) }
2024-11-11T19:03:16.562624Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    send    frame=Data { stream_id: StreamId(27), flags: (0x1: END_STREAM) }
2024-11-11T19:03:16.562702Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    received    frame=Data { stream_id: StreamId(27) }
2024-11-11T19:03:16.562906Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=05a5b63eac4c333c926126224ad72546}:Connection{peer=Client}    received    frame=Data { stream_id: StreamId(27), flags: (0x1: END_STREAM) }
2024-11-11T19:03:16.562945Z    info    access    connection complete    src.addr=100.100.14.218:44567 src.workload="authentication-service-79d5d9694c-w7vj5" src.namespace="light-grass" src.identity="spiffe://cluster.local/ns/light-grass/sa/authentication-service" dst.addr=100.100.8.138:15008 dst.hbone_addr=100.100.8.138:3306 dst.service="authentication-service-mysql.light-grass.svc.cluster.local" dst.workload="authentication-service-mysql-0" dst.namespace="light-grass" dst.identity="spiffe://cluster.local/ns/light-grass/sa/authentication-service-mysql" direction="outbound" bytes_sent=0 bytes_recv=78 duration="0ms"
2024-11-11T19:03:16.562954Z    debug    proxy::outbound:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=1ccdf79bfc148d161054e1a4179b8c49}    connection completed    component="outbound" dur=922.168µs
2024-11-11T19:03:16.563996Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    send    frame=Data { stream_id: StreamId(15) }
2024-11-11T19:03:16.567119Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    send    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:16.567210Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    received    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:16.567220Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    send    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:16.567930Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:Connection{peer=Server}    received    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.302852Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-redis-master-0}:Connection{peer=Server}    send    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.302866Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=da56dbfda709e42de6c7d34382ed6a61}:Connection{peer=Client}    send    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.302991Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-redis-master-0}:Connection{peer=Server}    received    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.302995Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=da56dbfda709e42de6c7d34382ed6a61}:Connection{peer=Client}    received    frame=Ping { ack: false, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.303002Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-redis-master-0}:Connection{peer=Server}    send    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.303002Z    debug    h2::codec::framed_write:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=da56dbfda709e42de6c7d34382ed6a61}:Connection{peer=Client}    send    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.303055Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-79d5d9694c-w7vj5}:outbound{id=da56dbfda709e42de6c7d34382ed6a61}:Connection{peer=Client}    received    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }
2024-11-11T19:03:19.303088Z    debug    h2::codec::framed_read:proxy{wl=light-grass/authentication-service-redis-master-0}:Connection{peer=Server}    received    frame=Ping { ack: true, payload: [59, 124, 219, 122, 11, 135, 22, 180] }

One further observation - the issue resolved itself as soon as I restarted the istio-cni-node pod.

@jaellio Yes, some nodes are fine while others are not. Config on the nodes is all the same. I am thinking more and more it feels like a race condition. It’s hard to replicate but after a few tries switching between 1.23.3 and 1.24.0 I can get it to happen.

Oh and to clarify, from ztunnel’s standpoint, it thinks everything is fine. Ztunnel pods are all healthy, but istio-cni logs 500s.