cilium: Nodeport is not accessible after restarting os network service

Is there an existing issue for this?

• I have searched the existing issues

What happened?

1. In this environment: Kubernetes version: 1.24.8 Cilium version: 1.12.4 OS: Ubuntu 22.4 IPAM mode: AWS ENS

2 with this network related configuration: /etc/systemd/netword.conf [Network] ManageForeignRoutes=no ManageForeignRoutingPolicyRules=no

[DHCPv4] #DUIDType=vendor #DUIDRawData=

[DHCPv6] #DUIDType=vendor #DUIDRawData=

/etc/systemd/network/99-unmanaged-devices.network

[Match] Name=ens[012346789]*

[Link] Unmanaged=yes

3 After running following commands: systemctl daemon-reload systemctl restart systemd-networkd

4 nodeport cannot be accessed from outside of Kubernetes cluster, but accessible inside kubernetes cluster

perhaps it is related to bug report: https://github.com/cilium/cilium/issues/18706

Cilium Version

Cilium version: 1.12.4

Kernel Version

5.15.0-1031-aws #35-Ubuntu SMP Fri Feb 10 02:07:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

1.24.8

Sysdump

🔍 Collecting Hubble flows from Cilium pods ⚠️ The following tasks failed, the sysdump may be incomplete: ⚠️ [11] Collecting Cilium egress NAT policies: failed to collect Cilium egress NAT policies: the server could not find the requested resource (get ciliumegressnatpolicies.cilium.io) ⚠️ [12] Collecting Cilium local redirect policies: failed to collect Cilium local redirect policies: the server could not find the requested resource (get ciliumlocalredirectpolicies.cilium.io) ⚠️ [17] Collecting CiliumClusterwideEnvoyConfigs: failed to collect CiliumClusterwideEnvoyConfigs: the server could not find the requested resource (get ciliumclusterwideenvoyconfigs.cilium.io) ⚠️ [18] Collecting CiliumEnvoyConfigs: failed to collect CiliumEnvoyConfigs: the server could not find the requested resource (get ciliumenvoyconfigs.cilium.io) ⚠️ hubble-flows-cilium-f5n7x: failed to collect hubble flows for “cilium-f5n7x” in namespace “kube-system”: command terminated with exit code 1: failed to connect to ‘unix:///var/run/cilium/hubble.sock’: connection error: desc = “transport: error while dialing: dial unix /var/run/cilium/hubble.sock: connect: no such file or directory”

⚠️ hubble-flows-cilium-kz8hc: failed to collect hubble flows for “cilium-kz8hc” in namespace “kube-system”: command terminated with exit code 1: failed to connect to ‘unix:///var/run/cilium/hubble.sock’: connection error: desc = “transport: error while dialing: dial unix /var/run/cilium/hubble.sock: connect: no such file or directory”

⚠️ hubble-flows-cilium-l2v8b: failed to collect hubble flows for “cilium-l2v8b” in namespace “kube-system”: command terminated with exit code 1: failed to connect to ‘unix:///var/run/cilium/hubble.sock’: connection error: desc = “transport: error while dialing: dial unix /var/run/cilium/hubble.sock: connect: no such file or directory”

⚠️ hubble-flows-cilium-ldwbh: failed to collect hubble flows for “cilium-ldwbh” in namespace “kube-system”: command terminated with exit code 1: failed to connect to ‘unix:///var/run/cilium/hubble.sock’: connection error: desc = “transport: error while dialing: dial unix /var/run/cilium/hubble.sock: connect: no such file or directory”

⚠️ hubble-flows-cilium-mgjxp: failed to collect hubble flows for “cilium-mgjxp” in namespace “kube-system”: command terminated with exit code 1: failed to connect to ‘unix:///var/run/cilium/hubble.sock’: connection error: desc = “transport: error while dialing: dial unix /var/run/cilium/hubble.sock: connect: no such file or directory”

⚠️ hubble-flows-cilium-sng9j: failed to collect hubble flows for “cilium-sng9j” in namespace “kube-system”: command terminated with exit code 1: failed to connect to ‘unix:///var/run/cilium/hubble.sock’: connection error: desc = “transport: error while dialing: dial unix /var/run/cilium/hubble.sock: connect: no such file or directory”

⚠️ Please note that depending on your Cilium version and installation options, this may be expected 🗳 Compiling sysdump

Relevant log output

Mar 27 07:07:17 ip-172-27-17-122 systemd[1]: Reloading.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: systemd-networkd-wait-online.service: Deactivated successfully.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Stopped Wait for Network to be Configured.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Stopping Wait for Network to be Configured...
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Stopping Network Configuration...
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[435]: ens5: DHCPv6 lease lost
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: systemd-networkd.service: Deactivated successfully.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Stopped Network Configuration.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Started ntp-systemd-netif.service.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Starting Network Configuration...
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: ntp-systemd-netif.service: Deactivated successfully.
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc25817e7e42ca: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc25817e7e42ca: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxcf5831060de8b: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxcf5831060de8b: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc882cebefac47: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc882cebefac47: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc834974b29ca7: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc834974b29ca7: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: cilium_host: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: cilium_host: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: cilium_net: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: cilium_net: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: ens5: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: ens5: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lo: Link UP
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lo: Gained carrier
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc25817e7e42ca: Gained IPv6LL
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxcf5831060de8b: Gained IPv6LL
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc882cebefac47: Gained IPv6LL
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: lxc834974b29ca7: Gained IPv6LL
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: cilium_host: Gained IPv6LL
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: cilium_net: Gained IPv6LL
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: ens5: Gained IPv6LL
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: Enumeration completed
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Started Network Configuration.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Starting Wait for Network to be Configured...
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd[4089]: ens5: DHCPv4 address 172.27.17.122/20 via 172.27.16.1
Mar 27 07:07:27 ip-172-27-17-122 dbus-daemon[475]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.88' (uid=100 pid=4089 comm="/lib/systemd/systemd-networkd " label="unconfined")
Mar 27 07:07:27 ip-172-27-17-122 systemd-networkd-wait-online[4090]: managing: ens5
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Started ntp-systemd-netif.service.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Starting Hostname Service...
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Finished Wait for Network to be Configured.
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: ntp-systemd-netif.service: Deactivated successfully.
Mar 27 07:07:27 ip-172-27-17-122 dbus-daemon[475]: [system] Successfully activated service 'org.freedesktop.hostname1'
Mar 27 07:07:27 ip-172-27-17-122 systemd[1]: Started Hostname Service.

Anything else?

No response

Code of Conduct

• I agree to follow this project’s Code of Conduct