kubernetes: kubelet's MASQUERADE support is insufficient

Today kubelet sets up an iptables MASQUERADE rule for any traffic destined for anything except 10.0.0.0/8. This is close, but not even correct on GCE, and certainly not right elsewhere.

First GCE. We probably want something like:

iptables -t nat -N KUBE-IPMASQ
iptables -t nat -A KUBE-IPMASQ -d 10.0.0.0/8 -j RETURN
iptables -t nat -A KUBE-IPMASQ -d 172.16.0.0/12 -j RETURN
iptables -t nat -A KUBE-IPMASQ -d 192.168.0.0/16 -j RETURN
iptables -t nat -A KUBE-IPMASQ -j MASQUERADE
iptables -t nat -I POSTROUTING -j KUBE-IPMASQ

This catches all traffic to RFC1918 ranges and masquerades it. We can probably optimize with CONNMARK or something so we only consider packets from containers. This is probably still imperfect, but better, for lack of project-wide NAT for egress.

For other environments, we really have no idea what the correct policy for this is. It is closer to “your nodes must handle this” than “we can handle this for you”. It’s debatable whether we should even try.

Either: a) We teach kubelet a lot more and let people pass flags to nearly=arbitrarily configure this b) We tell people to configure this as part of their node setup

This popped up when I realized GKE allows users to set up 172.* clusters - any traffic between containers in one of these will get masqueraded - not correct behavior!! This is not a huge deal right now because kube-proxy has the same effect when traversing services. As we fix kube-proxy in the wake of 1.0, masquerade will be a bigger deal, especially for micro-segmenting.

Additional considerations: VPNs have bizarre and very custom needs. Every such thing has an as yet unmeasured perf implication. This also pops up in our GCE firewall thing that @ArtfulCoder is working on.

@dchen1107 @alex-mohr