cilium: CI: Conformance E2E: client-egress-l7-named-port/pod-to-pod: command terminated with exit code 28 (timeout)

CI failure

Matrix entry: (5, 5.15-20230420.212204, iptables, true, disabled, dsr, true, true, true)

ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7-named-port/pod-to-pod]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-0: cilium-test/client-6b4b857d98-hnf49 (10.244.1.76) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (10.244.2.69:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-0: cilium-test/client-6b4b857d98-hnf49 (fd00:10:244:1::47c6) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (fd00:10:244:2::294e:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-1: cilium-test/client-6b4b857d98-hnf49 (10.244.1.76) -> cilium-test/echo-same-node-7c978647dc-xr2kt (10.244.1.111:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-1: cilium-test/client-6b4b857d98-hnf49 (fd00:10:244:1::47c6) -> cilium-test/echo-same-node-7c978647dc-xr2kt (fd00:10:244:1::19cd:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-646b88fb9b-x8j2f (10.244.1.44) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (10.244.2.69:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://10.244.2.69:8080/" failed: command terminated with exit code 28
  ℹ️  curl output:


📋 Test Report
connectivity test failed: 1 tests failed
❌ 1/53 tests failed (1/575 actions), 8 tests skipped, 0 scenarios skipped:
Test [client-egress-l7-named-port]:
  ❌ client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-646b88fb9b-x8j2f (10.244.1.44) -> cilium-test/echo-other-node-8f5d547bf-sgh62 (10.244.2.69:8080)
Error: Process completed with exit code 1.

Looks similar to https://github.com/cilium/cilium/issues/27672 but not related to multicluster.

From a quick sysdump check, there seems to be something wrong with Envoy and the http conneciton:

$ cat hubble-flows-cilium-* | hubble observe --input-file - --numeric --to-ip 10.244.2.69 --port 8080 --from-ip 10.244.1.44 --type l7
Aug 28 17:16:59.869: 10.244.1.44:44528 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/)
Aug 28 17:17:21.203: 10.244.1.44:54076 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/)
Aug 28 17:15:50.297: 10.244.1.44:37878 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/public)
Aug 28 17:15:50.377: 10.244.1.44:45332 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request DROPPED (HTTP/1.1 GET http://10.244.2.69:8080/private)
Aug 28 17:15:50.453: 10.244.1.44:45348 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/private)
Aug 28 17:16:18.302: 10.244.1.44:45148 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/public)
Aug 28 17:16:18.375: 10.244.1.44:45162 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request DROPPED (HTTP/1.1 GET http://10.244.2.69:8080/private)
Aug 28 17:16:18.445: 10.244.1.44:45166 (ID:11576) -> 10.244.2.69:8080 (ID:25450) http-request FORWARDED (HTTP/1.1 GET http://10.244.2.69:8080/private)

cilium-sysdump-5-final.zip

About this issue

Original URL
State: open
Created 10 months ago
Comments: 32 (32 by maintainers)

Commits related to this issue

connectivity: Add retries to PodToPod curl commands This commit adds retries to the curl commands in the PodToPod scenario. The goal of adding this change is to try and address CI flakes which may be... — committed to learnitall/cilium-cli by learnitall 10 months ago
connectivity: Add retries to PodToPod curl commands This commit adds retries to the curl commands in the PodToPod scenario. The goal of adding this change is to try and address CI flakes which may be... — committed to learnitall/cilium-cli by learnitall 10 months ago
connectivity: Add retries to PodToPod curl commands This commit adds retries to the curl commands in the PodToPod scenario. The goal of adding this change is to try and address the CI flake cilium/ci... — committed to learnitall/cilium-cli by learnitall 10 months ago
connectivity: Add retries to PodToPod curl commands This commit adds retries to the curl commands in the PodToPod scenario. The goal of adding this change is to try and address the Cilium CI flake ci... — committed to learnitall/cilium-cli by learnitall 10 months ago

Most upvoted comments

This is now also happening on ci-clustermesh.

lmb on Nov 29, 2023

Another hit here:

[=] Test [client-egress-l7-named-port]
.............
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-only-dns' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  ℹ️  📜 Applying CiliumNetworkPolicy 'client-egress-l7-http-named-port' to namespace 'cilium-test'..
  [-] Scenario [client-egress-l7-named-port/pod-to-world]
  [.] Action [client-egress-l7-named-port/pod-to-world/http-to-google.com-0: cilium-test/client-78f9dffc84-5l4h2 (10.242.1.254) -> google.com-http (google.com:80)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-google.com-0: cilium-test/client-78f9dffc84-5l4h2 (10.242.1.254) -> google.com-https (google.com:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-google.com-index-0: cilium-test/client-78f9dffc84-5l4h2 (10.242.1.254) -> google.com-https-index (google.com:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/http-to-google.com-1: cilium-test/client2-59b578d4bb-f78g6 (10.242.1.174) -> google.com-http (google.com:80)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-google.com-1: cilium-test/client2-59b578d4bb-f78g6 (10.242.1.174) -> google.com-https (google.com:443)]
  [.] Action [client-egress-l7-named-port/pod-to-world/https-to-google.com-index-1: cilium-test/client2-59b578d4bb-f78g6 (10.242.1.174) -> google.com-https-index (google.com:443)]
  [-] Scenario [client-egress-l7-named-port/pod-to-pod]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-0: cilium-test/client-78f9dffc84-5l4h2 (10.242.1.254) -> cilium-test/echo-same-node-79d996cb79-8whls (10.242.1.250:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-0: cilium-test/client-78f9dffc84-5l4h2 (fd00:10:242:1::bd80) -> cilium-test/echo-same-node-79d996cb79-8whls (fd00:10:242:1::f461:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-1: cilium-test/client-78f9dffc84-5l4h2 (10.242.1.254) -> cilium-test/echo-other-node-9746df45c-6xxbk (10.244.1.216:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-1: cilium-test/client-78f9dffc84-5l4h2 (fd00:10:242:1::bd80) -> cilium-test/echo-other-node-9746df45c-6xxbk (fd00:10:244:1::cd09:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-2: cilium-test/client2-59b578d4bb-f78g6 (10.242.1.174) -> cilium-test/echo-same-node-79d996cb79-8whls (10.242.1.250:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv6-2: cilium-test/client2-59b578d4bb-f78g6 (fd00:10:242:1::6c1e) -> cilium-test/echo-same-node-79d996cb79-8whls (fd00:10:242:1::f461:8080)]
  [.] Action [client-egress-l7-named-port/pod-to-pod/curl-ipv4-3: cilium-test/client2-59b578d4bb-f78g6 (10.242.1.174) -> cilium-test/echo-other-node-9746df45c-6xxbk (10.244.1.216:8080)]
  ❌ command "curl -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code} --silent --fail --show-error --output /dev/null --connect-timeout 2 --max-time 10 http://10.244.1.216:8080/" failed: command terminated with exit code 28

Sysdump: cilium-sysdumps (2).zip

pippolo84 on Oct 26, 2023

I think that would make sense. I’m going to do some more digging through the sysdumps to double check if this would help and then get started on an implementation.

learnitall on Sep 8, 2023