Flatcar: Upgrade to systemd 243+ breaks pod networking with AWS CNI due to veth MAC Address getting overwritten

I’ve applied the fix to all flatcar branches. I’ve verified that the test case provided by @dvulpe passes with this fix applied.

This fix will be included in the next set of flatcar releases.

I’ve tested this and verified that indeed the MACAdressPolicy is the likely culprit. To a broken node, I added:

core@ip-192-168-4-227 ~ $ cat /etc/systemd/network/50-veth.link 
[Match]
Driver=veth

[Link]
MACAddressPolicy=none

And after rebooting, new virtual interfaces work correctly. We had dealt with a similar issue for flannel here: https://github.com/kinvolk/coreos-overlay/pull/282. In my tests, I also tried matching by name (eni*) and it also solved the problem.

As with in the linked flannel issue, the problem is visible when using ip monitor all and catching the interface getting created:

[LINK]4: eni4bd1086a8e2@if3: <BROADCAST,MULTICAST> mtu 9001 qdisc noop state DOWN group default 
    link/ether fe:f3:b1:0e:64:f5 brd ff:ff:ff:ff:ff:ff link-netnsid 0
[LINK]4: eni4bd1086a8e2@if3: <BROADCAST,MULTICAST> mtu 9001 qdisc noop state DOWN group default 
    link/ether 0e:33:4a:64:18:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0
[LINK]4: eni4bd1086a8e2@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default 
    link/ether 0e:33:4a:64:18:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0

We have a few files that instruct systemd not to manage links for various CNIs, we do this by matching by interface names in the .network files, so we need to keep listing more and more with each different CNI. And also, at least in the case of flannel it was not enough, to avoid the mac address being managed we had to add the explicit .link file with the MacAddressPolicy=none setting.

These are the network files that currently state links should be unmanaged:

core@ip-192-168-4-227 /usr/lib64/systemd/network $ grep -ir -b1 unmanaged .
./ipsec-vti.network-69-[Link]
./ipsec-vti.network:76:Unmanaged=yes
--
./yy-azure-sriov-coreos.network-556-[Link]
./yy-azure-sriov-coreos.network:563:Unmanaged=yes
--
./yy-azure-sriov.network-557-[Link]
./yy-azure-sriov.network:564:Unmanaged=yes
--
./calico.network-79-[Link]
./calico.network:86:Unmanaged=yes
--
./50-flannel.network-23-[Link]
./50-flannel.network:30:Unmanaged=yes
--
./cni.network-74-[Link]
./cni.network:81:Unmanaged=yes

I think we should add a file like the one I showed above (matching on the veth driver) to our default setup. Alternatively, if we think that’s too broad, we could also match by name.

Thanks @marga-kinvolk, I deployed the latest image today and the test i conducted before are now passing!

Hi! First of all, thanks Dan for the reproduction case. I’ve used it and was able to verify that indeed this breaks when switching from 2512 to 2605. BTW, the repro case uses AWS CNI, not Cillium, so it already confirms what Greg commented.

I spent quite a few hours trying to figure out what exactly the problem is, but I wasn’t able to find the root cause. In the repro case, when using 2605 the coredns pod is unable to send or receive packets. They get dropped, but I couldn’t find what’s causing the drop.

I tried comparing the sysctl values across both versions and overriding some of those that were different, but that had no effect. The generated firewall rules have a few differences, including a comment that says that the Cluster IP is not reachable, but it’s unclear whether the differences are cause or effect of the problem.

I can also confirm that the 2605 series has broken the AWS CNI. Pods with host networking (e.g. kube-proxy or the CNI daemonset itself) are able to communicate outside the node fine, but pods without host networking fail to connect to anything as if the packets are being dropped, whether the destination is in-cluster, out-of-cluster, or even on the same node. This includes calls to the kubernetes.default service that exposes the control plane in the cluster.

Interestingly after reboot this works, but as the pod get’s rescheduled (on the same node) with a different IP it stops working again.