cilium: ipv6 endpoint connectivity regression 1.13.6 -> 1.15.0-pre.0

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

On v1.13.6, all cilium connectivity test checks pass, including IPv6 with this helm config: https://github.com/ocf/kubernetes/blob/c8014d3fd2cf5d0be7a6892f475cd47f59be2b45/apps/cilium.py

1.13.6 connectivity test report 
ℹ️  Cilium version: 1.13.6
🏃 Running tests...
[=] Test [no-policies]
......................................
[=] Test [no-policies-extra]
................
[=] Test [allow-all-except-world]
..................
[=] Test [client-ingress]
..
[=] Test [client-ingress-knp]
..
[=] Test [allow-all-with-metrics-check]
........
[=] Test [all-ingress-deny]
........
[=] Test [all-ingress-deny-knp]
........
[=] Test [all-egress-deny]
................
[=] Test [all-egress-deny-knp]
................
[=] Test [all-entities-deny]
........
[=] Test [cluster-entity]
..
[=] Test [host-entity]
........
[=] Test [echo-ingress]
....
[=] Test [echo-ingress-knp]
....
[=] Test [client-ingress-icmp]
..
[=] Test [client-egress]
....
[=] Test [client-egress-knp]
....
[=] Test [client-egress-expression]
....
[=] Test [client-egress-expression-knp]
....
[=] Test [client-with-service-account-egress-to-echo]
....
[=] Test [client-egress-to-echo-service-account]
....
[=] Test [to-entities-world]
......
[=] Test [to-cidr-external]
....
[=] Test [to-cidr-external-knp]
....
[=] Test [echo-ingress-from-other-client-deny]
......
[=] Test [client-ingress-from-other-client-icmp-deny]
......
[=] Test [client-egress-to-echo-deny]
......
[=] Test [client-ingress-to-echo-named-port-deny]
....
[=] Test [client-egress-to-echo-expression-deny]
....
[=] Test [client-with-service-account-egress-to-echo-deny]
....
[=] Test [client-egress-to-echo-service-account-deny]
..
[=] Test [client-egress-to-cidr-deny]
....
[=] Test [client-egress-to-cidr-deny-default]
....
[=] Test [health]
....

[=] Skipping Test [north-south-loadbalancing]

[=] Skipping Test [pod-to-pod-encryption]

[=] Skipping Test [node-to-node-encryption]

[=] Skipping Test [north-south-loadbalancing-with-l7-policy]
[=] Test [echo-ingress-l7]
............
[=] Test [echo-ingress-l7-named-port]
............
[=] Test [client-egress-l7-method]
............
[=] Test [client-egress-l7]
..........
[=] Test [client-egress-l7-named-port]
..........

[=] Skipping Test [client-egress-l7-tls-deny-without-headers]

[=] Skipping Test [client-egress-l7-tls-headers]

[=] Skipping Test [client-egress-l7-set-header]

[=] Skipping Test [echo-ingress-auth-always-fail]

[=] Skipping Test [echo-ingress-mutual-auth-spiffe]

[=] Skipping Test [pod-to-ingress-service]

[=] Skipping Test [pod-to-ingress-service-deny-all]

[=] Skipping Test [pod-to-ingress-service-allow-ingress-identity]
[=] Test [dns-only]
..........
[=] Test [to-fqdns]
........

✅ All 42 tests (316 actions) successful, 12 tests skipped, 0 scenarios skipped.

With the same config on v1.15.0-pre.0, IPv6 connectivity checks no longer pass (and empirically, no IPv6 connections between pods work).

1.15.0 connectivity test report 
ℹ️  Cilium version: 1.15.0
🏃 Running tests...
[=] Test [no-policies]
....................
  [-] Scenario [no-policies/pod-to-world]
  [.] Action [no-policies/pod-to-world/http-to-one.one.one.one-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> one.one.one.one-http (one.one.one.one:80)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-index-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> one.one.one.one-https-index (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/http-to-one.one.one.one-1: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> one.one.one.one-http (one.one.one.one:80)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-1: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> one.one.one.one-https (one.one.one.one:443)]
  [.] Action [no-policies/pod-to-world/https-to-one.one.one.one-index-1: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> one.one.one.one-https-index (one.one.one.one:443)]
  [-] Scenario [no-policies/pod-to-host]
  [.] Action [no-policies/pod-to-host/ping-ipv4-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.7 (169.229.226.7:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.9 (169.229.226.9:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-2: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.8 (169.229.226.8:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-3: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.10 (169.229.226.10:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-4: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.7 (169.229.226.7:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-5: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.9 (169.229.226.9:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-6: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.8 (169.229.226.8:0)]
  [.] Action [no-policies/pod-to-host/ping-ipv4-7: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.10 (169.229.226.10:0)]
  [-] Scenario [no-policies/pod-to-external-workload]
  [-] Scenario [no-policies/pod-to-cidr]
  [.] Action [no-policies/pod-to-cidr/external-1111-0: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> external-1111 (1.1.1.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1111-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> external-1111 (1.1.1.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1001-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> external-1001 (1.0.0.1:443)]
  [.] Action [no-policies/pod-to-cidr/external-1001-1: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> external-1001 (1.0.0.1:443)]
  [-] Scenario [no-policies/pod-to-pod]
  [.] Action [no-policies/pod-to-pod/curl-ipv4-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (10.244.139.200:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv6-0: cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (2607:f140:8801:1::222:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::222]:8080" failed: command terminated with exit code 28
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action curl-ipv6-0
  📄 No flows recorded for peer cilium-test/echo-other-node-5cbfc6f76f-xrvxz during action curl-ipv6-0
  [.] Action [no-policies/pod-to-pod/curl-ipv4-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (10.244.140.252:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv6-1: cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (2607:f140:8801:1::3d1:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::3d1]:8080" failed: command terminated with exit code 7
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action curl-ipv6-1
  📄 No flows recorded for peer cilium-test/echo-same-node-6fc6fcfb7-9wvtq during action curl-ipv6-1
  [.] Action [no-policies/pod-to-pod/curl-ipv4-2: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (10.244.139.200:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv6-2: cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (2607:f140:8801:1::222:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::222]:8080" failed: command terminated with exit code 28
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action curl-ipv6-2
  📄 No flows recorded for peer cilium-test/echo-other-node-5cbfc6f76f-xrvxz during action curl-ipv6-2
  [.] Action [no-policies/pod-to-pod/curl-ipv4-3: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (10.244.140.252:8080)]
  [.] Action [no-policies/pod-to-pod/curl-ipv6-3: cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (2607:f140:8801:1::3d1:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::3d1]:8080" failed: command terminated with exit code 7
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action curl-ipv6-3
  📄 No flows recorded for peer cilium-test/echo-same-node-6fc6fcfb7-9wvtq during action curl-ipv6-3
  [-] Scenario [no-policies/client-to-client]
  [.] Action [no-policies/client-to-client/ping-ipv4-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/client2-764b565764-js5zx (10.244.140.141:0)]
  [.] Action [no-policies/client-to-client/ping-ipv6-0: cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346) -> cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d:0)]
  ❌ command "ping -c 1 -6 -W 2 -w 10 2607:f140:8801:1::33d" failed: command terminated with exit code 1
  ℹ️  ping output:
  
  
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action ping-ipv6-0
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action ping-ipv6-0
  [.] Action [no-policies/client-to-client/ping-ipv4-1: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/client-84bfddc76b-zpknp (10.244.140.83:0)]
  [.] Action [no-policies/client-to-client/ping-ipv6-1: cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d) -> cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346:0)]
  ❌ command "ping -c 1 -6 -W 2 -w 10 2607:f140:8801:1::346" failed: command terminated with exit code 1
  ℹ️  ping output:
  
  
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action ping-ipv6-1
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action ping-ipv6-1
  [-] Scenario [no-policies/pod-to-service]
  [.] Action [no-policies/pod-to-service/curl-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-other-node (echo-other-node:8080)]
  [.] Action [no-policies/pod-to-service/curl-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-same-node (echo-same-node:8080)]
  [.] Action [no-policies/pod-to-service/curl-2: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-other-node (echo-other-node:8080)]
  [.] Action [no-policies/pod-to-service/curl-3: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-same-node (echo-same-node:8080)]
  [-] Scenario [no-policies/pod-to-hostport]
  [.] Action [no-policies/pod-to-hostport/curl-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (169.229.226.7:40000)]
  [.] Action [no-policies/pod-to-hostport/curl-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (169.229.226.9:40000)]
  [.] Action [no-policies/pod-to-hostport/curl-2: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (169.229.226.7:40000)]
  [.] Action [no-policies/pod-to-hostport/curl-3: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (169.229.226.9:40000)]
[=] Test [no-policies-extra]
................
[=] Test [allow-all-except-world]
..
  ℹ️  📜 Applying CiliumNetworkPolicy 'allow-all-except-world' to namespace 'cilium-test'..
  [-] Scenario [allow-all-except-world/pod-to-external-workload]
  [-] Scenario [allow-all-except-world/pod-to-pod]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (10.244.139.200:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-0: cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (2607:f140:8801:1::222:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::222]:8080" failed: command terminated with exit code 28
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action curl-ipv6-0
  📄 No flows recorded for peer cilium-test/echo-other-node-5cbfc6f76f-xrvxz during action curl-ipv6-0
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (10.244.140.252:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-1: cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (2607:f140:8801:1::3d1:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::3d1]:8080" failed: command terminated with exit code 7
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action curl-ipv6-1
  📄 No flows recorded for peer cilium-test/echo-same-node-6fc6fcfb7-9wvtq during action curl-ipv6-1
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-2: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (10.244.139.200:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-2: cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d) -> cilium-test/echo-other-node-5cbfc6f76f-xrvxz (2607:f140:8801:1::222:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::222]:8080" failed: command terminated with exit code 28
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action curl-ipv6-2
  📄 No flows recorded for peer cilium-test/echo-other-node-5cbfc6f76f-xrvxz during action curl-ipv6-2
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv4-3: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (10.244.140.252:8080)]
  [.] Action [allow-all-except-world/pod-to-pod/curl-ipv6-3: cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d) -> cilium-test/echo-same-node-6fc6fcfb7-9wvtq (2607:f140:8801:1::3d1:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://[2607:f140:8801:1::3d1]:8080" failed: command terminated with exit code 7
  ℹ️  curl output:
  
  
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action curl-ipv6-3
  📄 No flows recorded for peer cilium-test/echo-same-node-6fc6fcfb7-9wvtq during action curl-ipv6-3
  [-] Scenario [allow-all-except-world/client-to-client]
  [.] Action [allow-all-except-world/client-to-client/ping-ipv4-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/client2-764b565764-js5zx (10.244.140.141:0)]
  [.] Action [allow-all-except-world/client-to-client/ping-ipv6-0: cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346) -> cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d:0)]
  ❌ command "ping -c 1 -6 -W 2 -w 10 2607:f140:8801:1::33d" failed: command terminated with exit code 1
  ℹ️  ping output:
  
  
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action ping-ipv6-0
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action ping-ipv6-0
  [.] Action [allow-all-except-world/client-to-client/ping-ipv4-1: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/client-84bfddc76b-zpknp (10.244.140.83:0)]
  [.] Action [allow-all-except-world/client-to-client/ping-ipv6-1: cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d) -> cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346:0)]
  ❌ command "ping -c 1 -6 -W 2 -w 10 2607:f140:8801:1::346" failed: command terminated with exit code 1
  ℹ️  ping output:
  
  
  📄 No flows recorded for peer cilium-test/client2-764b565764-js5zx during action ping-ipv6-1
  📄 No flows recorded for peer cilium-test/client-84bfddc76b-zpknp during action ping-ipv6-1
  [-] Scenario [allow-all-except-world/pod-to-service]
  [.] Action [allow-all-except-world/pod-to-service/curl-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-other-node (echo-other-node:8080)]
  [.] Action [allow-all-except-world/pod-to-service/curl-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/echo-same-node (echo-same-node:8080)]
  [.] Action [allow-all-except-world/pod-to-service/curl-2: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-same-node (echo-same-node:8080)]
  [.] Action [allow-all-except-world/pod-to-service/curl-3: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> cilium-test/echo-other-node (echo-other-node:8080)]
  [-] Scenario [allow-all-except-world/pod-to-host]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.7 (169.229.226.7:0)]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-1: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.9 (169.229.226.9:0)]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-2: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.8 (169.229.226.8:0)]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-3: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> 169.229.226.10 (169.229.226.10:0)]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-4: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.10 (169.229.226.10:0)]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-5: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.7 (169.229.226.7:0)]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-6: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.9 (169.229.226.9:0)]
  [.] Action [allow-all-except-world/pod-to-host/ping-ipv4-7: cilium-test/client2-764b565764-js5zx (10.244.140.141) -> 169.229.226.8 (169.229.226.8:0)]
  ℹ️  📜 Deleting CiliumNetworkPolicy 'allow-all-except-world' from namespace 'cilium-test'..
[=] Test [client-ingress]
.^CInterrupt received, cancelling tests...
.
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-ingress-from-client2' to namespace 'cilium-test'..
  [-] Scenario [client-ingress/client-to-client]
  [.] Action [client-ingress/client-to-client/ping-ipv4-0: cilium-test/client-84bfddc76b-zpknp (10.244.140.83) -> cilium-test/client2-764b565764-js5zx (10.244.140.141:0)]
  ℹ️  unable to extract exit code from error: context canceled
  [.] Action [client-ingress/client-to-client/ping-ipv6-0: cilium-test/client-84bfddc76b-zpknp (2607:f140:8801:1::346) -> cilium-test/client2-764b565764-js5zx (2607:f140:8801:1::33d:0)]
  🟥 Skipping command execution: context canceled
  ℹ️  📜 Deleting CiliumNetworkPolicy 'client-ingress-from-client2' from namespace 'cilium-test'..
connectivity test failed: context canceled

Cilium Version

v1.15.0-pre.0

Kernel Version

Linux adenine 6.3.10 #1-NixOS SMP PREEMPT_DYNAMIC Wed Jun 28 09:14:25 UTC 2023 x86_64 GNU/Linux

Kubernetes Version

Server Version: v1.25.3

Sysdump

This sysdump is from the broken v1.15.0-pre.0 install…

https://www.ocf.berkeley.edu/~njha/tmp/cilium-sysdump-20230902-132916.zip

Relevant log output

(nothing looks suspicious from the logs i looked at, see sysdump for all logs)

Anything else?

No response

Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Reactions: 3
  • Comments: 20 (10 by maintainers)

Commits related to this issue

Most upvoted comments

Confirmed that the patch fixes this! 🎉

+2
ethanwu10 on Oct 6, 2023

I’ve spent a few afternoons trying to reproduce this on kind, but no luck (this requires tunneling to be disabled, which also breaks ipv4 for me). Don’t have the bandwidth to investigate more at the moment, so will unassign for now.

Clearly there’s an issue with the commit found in the bisect, but there’s nothing that sticks out. BPF_V6 loads into d1/d2 instead of p1-4. There are other users of BPF_V6 that seem to function as expected, but on the other hand, both nodeport and ndp proxying use ROUTER_IP, which we’re both receiving reports from. ROUTER_IP is set using DEFINE_IPV6 in production code.

Anyone up for trying to reproduce this on Kind? That would make this much easier to debug.

+2
ti-mo on Sep 14, 2023

@ethanwu10 Thanks for the feedback and the help investigating!

+1
ti-mo on Oct 6, 2023

The problem is that ffff:ffff:8801:0001:0000:0000:0000:0204 is not the right ROUTER_IP; in the node config, it is defined as 2607:f140:8801:..., however in the loaded program (the dump was obtained via bpftool), router is loaded with ffff:ffff:8801:.... target is the correct IP (the actual ICMP packet is for the right address), however the BPF program contains the wrong IP for router (i.e. router should contain 2607:f140:8801:...).

So, somehow either the node_config.h that I saw was not the contents that was used during compilation, or something goes wrong in the process of loading the program to the kernel that caused the constant to get changed (causing the top 4 bytes to get replaced with ff).

I’ll edit this with the dump of the .os from the broken commit once I get them. I checked the ELF I have right now from 1.13.6 and they seem corrupted though (program header is zero length and section header points to garbage).

ELFs in 1.14 are fine, but the .data section contains the wrong constants:

0000000000000030 <ROUTER_IP_1>:
       6:	ff ff ff ff 88 01 00 01	<unknown>

0000000000000038 <ROUTER_IP_2>:
       7:	00 00 00 00 00 00 02 04	<unknown>

(ROUTER_IP_1 should be 26 07 f1 40 88 01 00 01) - this behavior happens in both the template ELF (in templates/) and the templated ELFs (in 1234/bpf_lxc.o) Meanwhile, /var/run/cilium/state/globals/node_config.h is still correct:

DEFINE_IPV6(ROUTER_IP, 0x26, 0x7, 0xf1, 0x40, 0x88, 0x1, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0x4);

Full .data dump

It seems like HOST_IP is also similarly affected - the top 4 bytes are also replaced with ff (they were also 2607:f140 originally)

+1
ethanwu10 on Oct 3, 2023
Read more comments on GitHub
← cilium: IPV6 access to node is lost after installing cilium with ipv6 enabled
cilium: Jumbo payload failed in cilium v1.5.6 and after →