cilium: [Cilium 1.12.1][Kubernetes 1.25.0][sig-network] HostPort validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

When running conformance test with sonobuoy as below, it passed for Kubernetes 1.22/1.23/1.24 but only failed with 1.25.0-rc.0/1.25.0-rc.1/1.25.0:

sonobuoy run --e2e-focus="HostPort.*\[Conformance\].*"

Most setup is referencing https://github.com/cilium/cilium/blob/d940f795ad998e5467d214454d94d27f0ac27fce/test/kubernetes-test.sh#L11-L31

Just updated as --set kubeProxyReplacement=partial and --set cni.chainingMode=none according to 1.12.x upgrade suggestion.

Cilium Version

Cilium 1.12.1

root@shahqu9veeth-1:~# cilium version
cilium-cli: 0.12.1+2022081829e62502 compiled with go1.19 on linux/amd64
cilium image (default): v1.12.0
cilium image (stable): v1.12.1
cilium image (running): v1.12.1

Deploy as below:

helm template cilium cilium/cilium \
    --version 1.12.1 \
    --namespace kube-system \
    --set bpf.preallocateMaps=false \
    --set cleanBpfState=false \
    --set cleanState=false \
    --set cluster.id=0 \
    --set cluster.name=4e8b0505-4c52-57ab-a7f4-481e7ed3a2e3 \
    --set cni.binPath=/usr/libexec/cni \
    --set cni.chainingMode=none \
    --set cni.exclusive=true \
    --set enableIPv4Masquerade=true  \
    --set enableIPv6Masquerade=false \
    --set externalIPs.enabled=true \
    --set hostPort.enabled=true \
    --set hostServices.enabled=true \
    --set hubble.enabled=false \
    --set ipam.mode=cluster-pool \
    --set ipam.operator.clusterPoolIPv4MaskSize=24 \
    --set ipam.operator.clusterPoolIPv4PodCIDRList=10.233.64.0/18 \
    --set ipv4.enabled=true \
    --set ipv6.enabled=false \
    --set kubeProxyReplacement=partial \
    --set nodePort.enabled=true \
    --set nodeinit.enabled=true \
    --set nodeinit.securityContext.privileged=true \
    --set operator.replicas=1 \
    --set operator.securityContext.privileged=true \
    --set securityContext.privileged=true \
    --set sessionAffinity=true \
    --set sleepAfterInit=false \
    --set tunnel=vxlan \
    | yq . > cilium.yml
kubectl apply -Rf cilium.yml

With correct status:

root@shahqu9veeth-1:~# cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:         OK
 \__/¯¯\__/    Operator:       OK
 /¯¯\__/¯¯\    Hubble:         disabled
 \__/¯¯\__/    ClusterMesh:    disabled
    \__/

Deployment        cilium-operator    Desired: 1, Ready: 1/1, Available: 1/1
DaemonSet         cilium             Desired: 3, Ready: 3/3, Available: 3/3
Containers:       cilium             Running: 3
                  cilium-operator    Running: 1
Cluster Pods:     3/3 managed by Cilium
Image versions    cilium             quay.io/cilium/cilium:v1.12.1: 3
                  cilium-operator    quay.io/cilium/operator-generic:v1.12.1: 1

Kernel Version

root@shahqu9veeth-1:/tmp# uname -a
Linux shahqu9veeth-1 5.15.0-46-generic #49-Ubuntu SMP Thu Aug 4 18:03:25 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

root@shahqu9veeth-1:~# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T22:40:33Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:38:15Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}

root@shahqu9veeth-1:~# kubectl get node -o wide
NAME             STATUS   ROLES           AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
shahqu9veeth-1   Ready    control-plane   2d23h   v1.25.0   192.168.121.234   <none>        Ubuntu 22.04.1 LTS   5.15.0-46-generic   cri-o://1.25.0
shahqu9veeth-2   Ready    control-plane   2d23h   v1.25.0   192.168.121.60    <none>        Ubuntu 22.04.1 LTS   5.15.0-46-generic   cri-o://1.25.0
shahqu9veeth-3   Ready    <none>          2d23h   v1.25.0   192.168.121.37    <none>        Ubuntu 22.04.1 LTS   5.15.0-46-generic   cri-o://1.25.0

Sysdump

cilium-sysdump-20220824-055719.zip

Relevant log output

root@shahqu9veeth-1:~# cat /tmp/plugins/e2e/results/global/e2e.log
I0824 05:30:53.797193      13 e2e.go:116] Starting e2e run "9c9af485-eb4b-45bd-b902-c6c6d4a33f22" on Ginkgo node 1
Aug 24 05:30:54.044: INFO: Enabling in-tree volume drivers
Running Suite: Kubernetes e2e suite - /usr/local/bin
====================================================
Random Seed: 1661319052 - will randomize all specs

Will run 1 of 7067 specs
------------------------------
[SynchronizedBeforeSuite] 
test/e2e/e2e.go:76
[SynchronizedBeforeSuite] TOP-LEVEL
  test/e2e/e2e.go:76
{"msg":"Test Suite starting","completed":0,"skipped":0,"failed":0}
Aug 24 05:30:54.910: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:30:54.957: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable
Aug 24 05:30:55.037: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Aug 24 05:30:55.125: INFO: 20 / 20 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Aug 24 05:30:55.125: INFO: expected 3 pod replicas in namespace 'kube-system', 3 are Running and Ready.
Aug 24 05:30:55.125: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
Aug 24 05:30:55.141: INFO: 3 / 3 pods ready in namespace 'kube-system' in daemonset 'cilium' (0 seconds elapsed)
Aug 24 05:30:55.141: INFO: 3 / 3 pods ready in namespace 'kube-system' in daemonset 'cilium-node-init' (0 seconds elapsed)
Aug 24 05:30:55.141: INFO: 3 / 3 pods ready in namespace 'kube-system' in daemonset 'kube-proxy' (0 seconds elapsed)
Aug 24 05:30:55.141: INFO: e2e test version: v1.25.0
Aug 24 05:30:55.145: INFO: kube-apiserver version: v1.25.0
[SynchronizedBeforeSuite] TOP-LEVEL
  test/e2e/e2e.go:76
Aug 24 05:30:55.152: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:30:55.173: INFO: Cluster IP family: ipv4
------------------------------
[SynchronizedBeforeSuite] PASSED [0.263 seconds]
[SynchronizedBeforeSuite] 
test/e2e/e2e.go:76

  Begin Captured GinkgoWriter Output >>
    [SynchronizedBeforeSuite] TOP-LEVEL
      test/e2e/e2e.go:76
    Aug 24 05:30:54.910: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
    Aug 24 05:30:54.957: INFO: Waiting up to 30m0s for all (but 0) nodes to be schedulable
    Aug 24 05:30:55.037: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
    Aug 24 05:30:55.125: INFO: 20 / 20 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
    Aug 24 05:30:55.125: INFO: expected 3 pod replicas in namespace 'kube-system', 3 are Running and Ready.
    Aug 24 05:30:55.125: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
    Aug 24 05:30:55.141: INFO: 3 / 3 pods ready in namespace 'kube-system' in daemonset 'cilium' (0 seconds elapsed)
    Aug 24 05:30:55.141: INFO: 3 / 3 pods ready in namespace 'kube-system' in daemonset 'cilium-node-init' (0 seconds elapsed)
    Aug 24 05:30:55.141: INFO: 3 / 3 pods ready in namespace 'kube-system' in daemonset 'kube-proxy' (0 seconds elapsed)
    Aug 24 05:30:55.141: INFO: e2e test version: v1.25.0
    Aug 24 05:30:55.145: INFO: kube-apiserver version: v1.25.0
    [SynchronizedBeforeSuite] TOP-LEVEL
      test/e2e/e2e.go:76
    Aug 24 05:30:55.152: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
    Aug 24 05:30:55.173: INFO: Cluster IP family: ipv4
  << End Captured GinkgoWriter Output
------------------------------
[ReportAfterEach] TOP-LEVEL
  test/e2e/e2e_test.go:142
S[ReportAfterEach] TOP-LEVEL
  test/e2e/e2e_test.go:142
S
------------------------------
[sig-network] HostPort
  validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]
  test/e2e/network/hostport.go:63
[BeforeEach] [sig-network] HostPort
  test/e2e/framework/framework.go:186
STEP: Creating a kubernetes client 08/24/22 05:30:55.573
Aug 24 05:30:55.577: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
STEP: Building a namespace api object, basename hostport 08/24/22 05:30:55.587
STEP: Waiting for a default service account to be provisioned in namespace 08/24/22 05:30:55.663
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace 08/24/22 05:30:55.668
[BeforeEach] [sig-network] HostPort
  test/e2e/network/hostport.go:49
[It] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]
  test/e2e/network/hostport.go:63
STEP: Trying to create a pod(pod1) with hostport 54323 and hostIP 127.0.0.1 and expect scheduled 08/24/22 05:30:55.696
Aug 24 05:30:55.724: INFO: Waiting up to 5m0s for pod "pod1" in namespace "hostport-9666" to be "running and ready"
Aug 24 05:30:55.731: INFO: Pod "pod1": Phase="Pending", Reason="", readiness=false. Elapsed: 7.905264ms
Aug 24 05:30:55.732: INFO: The phase of Pod pod1 is Pending, waiting for it to be Running (with Ready = true)
Aug 24 05:30:57.745: INFO: Pod "pod1": Phase="Pending", Reason="", readiness=false. Elapsed: 2.021464315s
Aug 24 05:30:57.745: INFO: The phase of Pod pod1 is Pending, waiting for it to be Running (with Ready = true)
Aug 24 05:30:59.743: INFO: Pod "pod1": Phase="Running", Reason="", readiness=true. Elapsed: 4.019272159s
Aug 24 05:30:59.743: INFO: The phase of Pod pod1 is Running (Ready = true)
Aug 24 05:30:59.743: INFO: Pod "pod1" satisfied condition "running and ready"
STEP: Trying to create another pod(pod2) with hostport 54323 but hostIP 192.168.121.234 on the node which pod1 resides and expect scheduled 08/24/22 05:30:59.743
Aug 24 05:30:59.760: INFO: Waiting up to 5m0s for pod "pod2" in namespace "hostport-9666" to be "running and ready"
Aug 24 05:30:59.766: INFO: Pod "pod2": Phase="Pending", Reason="", readiness=false. Elapsed: 5.911673ms
Aug 24 05:30:59.766: INFO: The phase of Pod pod2 is Pending, waiting for it to be Running (with Ready = true)
Aug 24 05:31:01.774: INFO: Pod "pod2": Phase="Pending", Reason="", readiness=false. Elapsed: 2.014328405s
Aug 24 05:31:01.775: INFO: The phase of Pod pod2 is Pending, waiting for it to be Running (with Ready = true)
Aug 24 05:31:03.774: INFO: Pod "pod2": Phase="Running", Reason="", readiness=true. Elapsed: 4.01365859s
Aug 24 05:31:03.774: INFO: The phase of Pod pod2 is Running (Ready = true)
Aug 24 05:31:03.774: INFO: Pod "pod2" satisfied condition "running and ready"
STEP: Trying to create a third pod(pod3) with hostport 54323, hostIP 192.168.121.234 but use UDP protocol on the node which pod2 resides 08/24/22 05:31:03.774
Aug 24 05:31:03.783: INFO: Waiting up to 5m0s for pod "pod3" in namespace "hostport-9666" to be "running and ready"
Aug 24 05:31:03.789: INFO: Pod "pod3": Phase="Pending", Reason="", readiness=false. Elapsed: 5.337807ms
Aug 24 05:31:03.789: INFO: The phase of Pod pod3 is Pending, waiting for it to be Running (with Ready = true)
Aug 24 05:31:05.809: INFO: Pod "pod3": Phase="Pending", Reason="", readiness=false. Elapsed: 2.025344755s
Aug 24 05:31:05.809: INFO: The phase of Pod pod3 is Pending, waiting for it to be Running (with Ready = true)
Aug 24 05:31:07.799: INFO: Pod "pod3": Phase="Running", Reason="", readiness=true. Elapsed: 4.015880921s
Aug 24 05:31:07.800: INFO: The phase of Pod pod3 is Running (Ready = true)
Aug 24 05:31:07.800: INFO: Pod "pod3" satisfied condition "running and ready"
Aug 24 05:31:07.808: INFO: Waiting up to 5m0s for pod "e2e-host-exec" in namespace "hostport-9666" to be "running and ready"
Aug 24 05:31:07.815: INFO: Pod "e2e-host-exec": Phase="Pending", Reason="", readiness=false. Elapsed: 6.777247ms
Aug 24 05:31:07.816: INFO: The phase of Pod e2e-host-exec is Pending, waiting for it to be Running (with Ready = true)
Aug 24 05:31:09.835: INFO: Pod "e2e-host-exec": Phase="Running", Reason="", readiness=true. Elapsed: 2.026835963s
Aug 24 05:31:09.835: INFO: The phase of Pod e2e-host-exec is Running (Ready = true)
Aug 24 05:31:09.835: INFO: Pod "e2e-host-exec" satisfied condition "running and ready"
STEP: checking connectivity from pod e2e-host-exec to serverIP: 127.0.0.1, port: 54323 08/24/22 05:31:09.845
Aug 24 05:31:09.846: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 --interface 192.168.121.234 http://127.0.0.1:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:09.846: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:09.847: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:09.847: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+--interface+192.168.121.234+http%3A%2F%2F127.0.0.1%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
STEP: checking connectivity from pod e2e-host-exec to serverIP: 192.168.121.234, port: 54323 08/24/22 05:31:10.476
Aug 24 05:31:10.476: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 http://192.168.121.234:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:10.476: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:10.479: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:10.479: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+http%3A%2F%2F192.168.121.234%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
Aug 24 05:31:10.901: INFO: Can not connect from e2e-host-exec to pod(pod2) to serverIP: 192.168.121.234, port: 54323
STEP: checking connectivity from pod e2e-host-exec to serverIP: 127.0.0.1, port: 54323 08/24/22 05:31:10.901
Aug 24 05:31:10.901: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 --interface 192.168.121.234 http://127.0.0.1:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:10.901: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:10.905: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:10.905: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+--interface+192.168.121.234+http%3A%2F%2F127.0.0.1%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
STEP: checking connectivity from pod e2e-host-exec to serverIP: 192.168.121.234, port: 54323 08/24/22 05:31:11.073
Aug 24 05:31:11.073: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 http://192.168.121.234:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:11.073: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:11.081: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:11.081: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+http%3A%2F%2F192.168.121.234%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
Aug 24 05:31:11.205: INFO: Can not connect from e2e-host-exec to pod(pod2) to serverIP: 192.168.121.234, port: 54323
STEP: checking connectivity from pod e2e-host-exec to serverIP: 127.0.0.1, port: 54323 08/24/22 05:31:11.205
Aug 24 05:31:11.205: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 --interface 192.168.121.234 http://127.0.0.1:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:11.205: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:11.206: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:11.207: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+--interface+192.168.121.234+http%3A%2F%2F127.0.0.1%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
STEP: checking connectivity from pod e2e-host-exec to serverIP: 192.168.121.234, port: 54323 08/24/22 05:31:12.43
Aug 24 05:31:12.431: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 http://192.168.121.234:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:12.431: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:12.432: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:12.432: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+http%3A%2F%2F192.168.121.234%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
Aug 24 05:31:12.738: INFO: Can not connect from e2e-host-exec to pod(pod2) to serverIP: 192.168.121.234, port: 54323
STEP: checking connectivity from pod e2e-host-exec to serverIP: 127.0.0.1, port: 54323 08/24/22 05:31:12.739
Aug 24 05:31:12.739: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 --interface 192.168.121.234 http://127.0.0.1:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:12.739: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:12.740: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:12.740: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+--interface+192.168.121.234+http%3A%2F%2F127.0.0.1%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
STEP: checking connectivity from pod e2e-host-exec to serverIP: 192.168.121.234, port: 54323 08/24/22 05:31:12.99
Aug 24 05:31:12.990: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 http://192.168.121.234:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:12.990: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:12.991: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:12.992: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+http%3A%2F%2F192.168.121.234%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
Aug 24 05:31:13.113: INFO: Can not connect from e2e-host-exec to pod(pod2) to serverIP: 192.168.121.234, port: 54323
STEP: checking connectivity from pod e2e-host-exec to serverIP: 127.0.0.1, port: 54323 08/24/22 05:31:13.113
Aug 24 05:31:13.113: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 --interface 192.168.121.234 http://127.0.0.1:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:13.113: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:13.121: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:13.121: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+--interface+192.168.121.234+http%3A%2F%2F127.0.0.1%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
STEP: checking connectivity from pod e2e-host-exec to serverIP: 192.168.121.234, port: 54323 08/24/22 05:31:14.004
Aug 24 05:31:14.004: INFO: ExecWithOptions {Command:[/bin/sh -c curl -g --connect-timeout 5 http://192.168.121.234:54323/hostname] Namespace:hostport-9666 PodName:e2e-host-exec ContainerName:e2e-host-exec Stdin:<nil> CaptureStdout:true CaptureStderr:true PreserveWhitespace:false Quiet:false}
Aug 24 05:31:14.004: INFO: >>> kubeConfig: /tmp/kubeconfig-2686308178
Aug 24 05:31:14.006: INFO: ExecWithOptions: Clientset creation
Aug 24 05:31:14.006: INFO: ExecWithOptions: execute(POST https://10.233.0.1:443/api/v1/namespaces/hostport-9666/pods/e2e-host-exec/exec?command=%2Fbin%2Fsh&command=-c&command=curl+-g+--connect-timeout+5+http%3A%2F%2F192.168.121.234%3A54323%2Fhostname&container=e2e-host-exec&container=e2e-host-exec&stderr=true&stdout=true)
Aug 24 05:31:14.301: INFO: Can not connect from e2e-host-exec to pod(pod2) to serverIP: 192.168.121.234, port: 54323
Aug 24 05:31:14.301: FAIL: Failed to connect to exposed host ports

root@kube22:/tmp# cat plugins/e2e/results/global/hostport-9666/e2e-host-exec/e2e-host-exec/logs.txt
Paused

root@kube22:/tmp# cat plugins/e2e/results/global/hostport-9666/pod1/agnhost/logs.txt
I0824 05:30:58.887510       1 log.go:195] Started HTTP server on port 8080
I0824 05:30:59.060701       1 log.go:195] GET /hostname
I0824 05:31:06.500004       1 log.go:195] GET /hostname
I0824 05:31:10.470642       1 log.go:195] GET /hostname
I0824 05:31:11.049708       1 log.go:195] GET /hostname
I0824 05:31:12.368963       1 log.go:195] GET /hostname
I0824 05:31:12.984214       1 log.go:195] GET /hostname
I0824 05:31:13.990417       1 log.go:195] GET /hostname

root@kube22:/tmp# cat plugins/e2e/results/global/hostport-9666/pod2/agnhost/logs.txt
I0824 05:31:02.355267       1 log.go:195] Started HTTP server on port 8080
I0824 05:31:03.037075       1 log.go:195] GET /hostname
I0824 05:31:10.396581       1 log.go:195] GET /hostname

root@kube22:/tmp# cat plugins/e2e/results/global/hostport-9666/pod3/agnhost/logs.txt
I0824 05:31:06.303797       1 log.go:195] Started HTTP server on port 8008
I0824 05:31:06.305332       1 log.go:195] Started UDP server on port  8080
I0824 05:31:07.130170       1 log.go:195] GET /hostname
I0824 05:31:14.223750       1 log.go:195] GET /hostname

Anything else?

image

Code of Conduct

  • I agree to follow this project’s Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 20 (13 by maintainers)

Commits related to this issue

Most upvoted comments

@aspsk @aanm any good news for this issue?

Hi @hswong3i

Just returned to this issue.

I am testing a change to fix the “localhost” part: https://github.com/cilium/cilium/compare/master...aspsk:cilium:aspsk/pr/allow-hostport-on-localhost

Also, I am going to work on the “TCP vs. UDP” issue next.

+3
aspsk on Sep 20, 2022

Is somebody working on this issue?

Not at the moment @DockToFuture However, we’re looking into if we can prioritize the https://github.com/cilium/cilium/issues/9207 issue (which will fix this one as well).

+2
aspsk on Mar 31, 2023

@tnorlin @hswong3i just FYI the reason why it works for @hswong3i is because you are running with “kubeProxyReplacement: partial” and @tnorlin is running with “kubeProxyReplacement: strict”.

With “kubeProxyReplacement: partial”, the test will pass because it’s using kube-proxy to perform the hostPort translation. @aspsk Is there a away to only disable the hostPort translation by Cilium when running in “kubeProxyReplacement: strict”?

+1
aanm on Mar 20, 2023

Should I raise a new issue or is there obvious mistakes with the flags/options?

@tnorlin Sorry, I’ve missed your question. The https://github.com/cilium/cilium/issues/9207 is still open, so you’re probably triggering it.

+1
aspsk on Mar 20, 2023

This is probably a duplicate of https://github.com/cilium/cilium/issues/14287 The sonobuoy test could (and should) fail for two reasons:

  1. hostport on 127.0.0.1 (see corresponding level=warning msg="The requested loopback address for hostIP (127.0.0.1) is not supported. Ignoring." hostIP=192.168.121.234 k8sNamespace=hostport-194 k8sPodName=pod1 podIPs="[]" subsys=k8s-watcher messages from the sysdump). This previously was bypassed by using portmap as described in this commit

  2. TCP vs. UDP. Previously this part would pass, but then we fixed the test itself. Presumably, this is exactly why it fails on k8s 1.25 and don’t fail on previous.

This is my next priority after the current issue, so I will get more info next week.

+1
aspsk on Aug 25, 2022

I had read about #9207 too, but this issue seems most likely due to some under layer Kubernetes 1.25 changes, because:

  1. Cilium < 1.12.0, I need to use chain mode in portmap for passing conformance test with Kubernetes < 1.25.0, as our document noted
  2. With >= Cilium 1.12.0 + Kubernetes < 1.25.0, chain mode could now update as none and still passing conformance test
  3. Now only failed with Cilium >= 1.12.1 + Kubernetes >= 1.25.0, and only failed with this hostport related test, no matter chain mode with portmap nor none
+1
hswong3i on Aug 25, 2022
Read more comments on GitHub
← cilium: Cilium 1.11 fails the kubernetes e2e test "[sig-network] Services should create endpoints for unready pods"
cilium: Cilium agent logs complain about `Unable to fetch kubernetes labels` →