metallb: Make ARP mode failover faster with virtual MACs

Is this a bug report or a feature request?:

Bug

What happened:

While writing the documentation for ARP mode, I realized that the way we’ve implemented the failover probably doesn’t work in all cases. The problem is that we are not using a single MAC address, so the flooding we use to try and update egress port maps on switches is useless.

As a reminder, our current behavior when a node is master elected is:

1. Respond to ARP who-has requests for service-ip with the node’s MAC address
2. Periodically flood ARP responses for service-ip with the node’s MAC address

The purpose of (1) is so that end hosts can resolve IP->MAC, to transmit the packets with the correct dst-mac+dst-ip. The purpose of (2) is to teach learning switches the MAC->egress-port mapping.

In a non-failover scenario, this works fine, and in fact in that scenario (2) is not needed because (1) indirectly teaches the switches in the path as well as the end-host.

In a failover scenario from node A to node B, this all falls apart. For new clients things work okay: they send an ARP request, receive a response that maps IP->node-B-MAC, and everything works fine. Existing clients have the old mapping of IP->node-A-MAC cached, so they keep transmitting to node A’s MAC address.

In theory, (2) is supposed to fix the second case. When node B becomes master, it floods a bunch of ARP responses that advertise a mapping of IP->node-B-MAC. End hosts ignore these (because they didn’t request that information). Switches listen to these messages, but their internal maps are MAC->egress-port, not IP->MAC or IP->egress-port. So, they happily update that node-B-MAC lives on port 42… And this does nothing for the traffic that is still going to node-A-MAC.

I made a mistake when I sketched out the design for ARP mode. Specifically, I forgot that VRRP defines a “virtual router MAC address”, and uses that MAC instead of the node’s MAC address for virtual router services.

So, when using VRRP, clients receive a mapping of IP->VRRP-MAC instead of IP->node-X-MAC, and switches learn a mapping of VRRP-MAC->port-42. When a failover happens, the new master floods ARP responses for VRRP-MAC, and this correctly updates the switch mapping to VRRP-MAC->port-50, and so all traffic, past and future, gets rerouted to the new master.

What does this mean for MetalLB?

1. The flooding behavior I insisted we implement is basically useless. It’s flooding irrelevant information to the switches, so we can just delete it 😦
2. ARP mode failover right now is very non-ideal. If you’re doing a graceful drain of a node, you should wait >60s after MetalLB leader failover before powering off the node. This should be long enough to let most clients refresh their ARP cache. Linux uses a 60s TTL, windows uses 15—45s. I can’t find reliable documentation about MacOS, but one place suggests it uses a 20 minute TTL, which is pretty ridiculous.
3. We should investigate implementing virtual MAC address support. This is probably going to be quite complex, because we have to modify the configuration of the network interface (otherwise the NIC will drop packets going to the virtual MAC), so this could have some pretty bad side-effects like split-brain if we get it wrong 😦

For 0.3, it’s probably okay to just document the limitations of ARP mode for now, unless we find a really easy way to implement virtual MACs without much complication. The current ARP mode is still very useful in general, it just has some poor behaviors during (hopefully rare) failovers. @miekg, wdyt?