cilium: Host network broken after one of the underlying interfaces of a bond goes down

I faced this today on a very simple Ubuntu 22.04 install - simply running sudo netplan apply manually was enough to break the host network after Cilium had been running. Setting

ManageForeignRoutes=no
ManageForeignRoutingPolicyRules=no

in /etc/systemd/networkd.conf did indeed fix it, so I wonder if we should document that?

We also hit the same issue with flatcar 3033.2.1 and systemd 249.4. It seems that systemd-networkd removed routing policy rules including local and the host couldn’t recognize localhost because of it. Cilium moves the policy rule for local when L7Poxy is enabled and systemd-networkd regards this local rule as a foreign routing policy and removes it.

$ ip rule list
32766:  from all lookup main
32767:  from all lookup default

network: drop unnecessary routing policy rules https://github.com/systemd/systemd/commit/0b81225e5791f660506f7db0ab88078cf296b771

For Flatcar Stable this works in /etc/systemd/networkd.conf:

[Network]
ManageForeignRoutes=no
ManageForeignRoutingPolicyRules=no

Then the additional IgnoreCarrierLoss=yes and KeepConfiguration=yes workarounds mentioned in https://github.com/cilium/cilium/issues/18706#issuecomment-1031470456 are not needed.

Does anyone know when this will be released?

This is part of quay.io/cilium/cilium:v1.14.0-snapshot.3 for testing and will go into 1.14.0.

https://github.com/cilium/cilium/releases/tag/v1.14.0-snapshot.3

I just ran into this problem, and it broke both IPv4 and IPv6. The solution in https://github.com/cilium/cilium/issues/18706#issuecomment-1031572546 did fix IPv4. It would have saved me a lot of time if this had been documented in an obvious place. Please consider documenting it in the installation manual, or better yet, fix it automatically.

yes it seems to have. That’s the default configuration into /etc/systemd/network.conf

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file, or by creating "drop-ins" in
# the networkd.conf.d/ subdirectory. The latter is generally recommended.
# Defaults can be restored by simply deleting this file and all drop-ins.
#
# See networkd.conf(5) for details.

[Network]
#SpeedMeter=no
#SpeedMeterIntervalSec=10sec
#ManageForeignRoutingPolicyRules=yes
#ManageForeignRoutes=yes
#RouteTable=

[DHCPv4]
#DUIDType=vendor
#DUIDRawData=

[DHCPv6]
#DUIDType=vendor
#DUIDRawData=

I do

sed -i /etc/systemd/networkd.conf -e 's/^#ManageForeignRoutingPolicyRules=yes$/ManageForeignRoutingPolicyRules=no/g'
sed -i /etc/systemd/networkd.conf -e 's/^#ManageForeignRoutes=yes$/ManageForeignRoutes=no/g'
systemctl restart systemd-networkd

Then setup cilium again and no more host network breaking during cilium network device creation

Ok, this manually brought it to working state:

sudo ip rule add from all fwmark 0x200/0xf00 lookup 2004 pref 9
sudo ip rule add from all fwmark 0xa00/0xf00 lookup 2005 pref 10
sudo ip rule add from all lookup local pref 100

Not sure if it’s easy to create a networkd unit that does the same. (If it is possible it would be a way to avoid requiring manual tweaking of global networkd settings if Cilium writes a networkd unit into the host’s /run to play well with networkd by default.)

You can use ManageForeignRoutingPolicyRules=no to protect policies. https://github.com/systemd/systemd/commit/d94dfe7053d49fa62c4bfc07b7f3fc2227c10aff

However, things are a little bit complicated because of this bug. This bug is backported to systems v249.5. With systemd v249.4, you need to set both IgnoreCarrierLoss=yes and KeepConfiguration=yes as well and never use networkctl reconfigure or networkctl reload. Or you need to create policy rules before networkd starts. I think the latter is difficult in our case.

Only ManageForeignRoutingPolicyRules=no is sufficient if you can use systemd v249.5.

Having same problem with:

[Network]
ManageForeignRoutes=no
ManageForeignRoutingPolicyRules=no

on Ubuntu Server 22.04.2 LTS:

$ systemd --version
systemd 249 (249.11-0ubuntu3.9)
$ uname -a
Linux srv-magic-master 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

with 1.14.0.snapshot.2

once I stopped k3s with k3s-killall.sh - network access is lost, but host still replies to pings

@borkmann would it make sense to have a preflight check for cilium around these settings, given that they can vary OS to OS by default and the combination of systemd and cilium configuration seem to be in conflict with eachother?

Good question, yes, I think it would make sense to have some automation on this regard to avoid users running into it. I haven’t checked yet how reliably one could detect systemd versions and/or whether older versions of systemd would just ignore some of the not-yet-supported networkd.conf settings. A preflight check seems sensible. cc @aanm

The preflight check is only intended for Cilium upgrades. I assume the issue can happen when the kernel upgrades but Cilium is not. Unless you were referring to a preflight inside the cilium-agent pod?

I was thinking of a start-up check. The issue is we faced was when switch maintenance, we lost an entire Kubernetes cluster because every node had lost host networking, and restarting was the fastest fix to restore service. In effect, these systemd-networkd + cilium configurations are dangerous for a production environment where I anticipated to have LACP up and working.

The networkd drop-ins do resolve my particular use case, however, I can’t be the only cilium user on a similar setup.

Thanks @ysksuzuki for the findings

Right, with Unmanaged=yes the rules are ignored and the only way for Cilium to tell networkd to preserve them is by putting them into an active .network unit (e.g., for a dummy interface)…

For reference, this is the translation into the networkd syntax:

[RoutingPolicyRule]
From=0.0.0.0/0
Table=local
Priority=100
[RoutingPolicyRule]
From=0.0.0.0/0
Table=2004
FirewallMark=512/3840
Priority=9
[RoutingPolicyRule]
From=0.0.0.0/0
Table=2005
FirewallMark=2560/3840
Priority=10

I think the networkd people had their reasons already from a “server” profile perspective. The way forward is that Flatcar, as a distro that expects Cilium to be running, would predefine the global setting like we do for similar cases. Maybe it makes sense for Cilium to add a check in the cilium install phase whether networkd is used and then recommending to change the default (plus having Unmanaged=yes is also a good idea even though it is only needed if people try to match too generically in their own networkd units as happend with Flatcar’s default unit).

On the topic of the disappearing lo entry without ManageForeignRoutingPolicyRules=no (on Flatcar Alpha), without Cilium running the down/up action of the underlying device has no impact and 0: from all lookup local stays in the list. With Cilium the rule is gone… Edit: Now I read Cilium moves the policy rule for local when L7Poxy is enabled and systemd-networkd regards this local rule as a foreign routing policy and removes it. again, that explains it.

These are the local addresses, pinging them gives packet loss while pinging 127.0.0.1 gives the permission denied issue as mentioned above.

Sure, I realize now that the quick-install.yaml used a hardcoded older rversion, will do it again.