moby: [Swarm-mode] Encrypted overlay traffic fails to transit NAT (AWS/GCE) on Debian kernel > 4.4

TL;DR: UDP checksum is wrong because ESP packets go through NAT. Configuring the VXLAN interfaces to not set the UDP checksum solves the issue.

My previous post on this issue contains a messy workaround while this post contains a real solution. It consists of configuring the VXLAN interfaces on the different nodes to not calculate the UDP checksum when transmitting packets. According to the ip-link man page such configuration is accomplished with the noudpcsum parameter. There are a two minor obstacles to carrying out this configuration: first, the VXLAN interfaces exist in a namespace automatically created by Docker, which we need to find, and second, the noudpcsum parameter cannot be changed so it is necessary to delete and recreate the interface.

1. Find the correct namespace

We will start by finding the namespace which contains the VXLAN interface which constitute the overlay network. For this we inspect the ingress network like so:

docker network inspect ingress

and note the “Id” parameter. Now we go to /var/run/docker/netns and look for the namespace that matches the “Id” of the ingress. On my nodes the ingress network has an “Id” starting with “de0d3f6935”. There is a corresponding namespace with filename /var/run/docker/netns/1-de0d3f6935, which we take note of.

2. Make the namespace accesible to ip netns

If you run ip netns list you will find that the namespaces created by Docker do not appear in the output. This is because they are not in the path where ip netns expects the namespaces, which is /var/run/netns. To solve this we create the /var/run/netns directory if necessary and create a soft link in that directory to the namespace found in step 1.

ln -s /var/run/docker/netns/1-de0d3f6935

Now ip netns list will show that namespace and, more importantly, we will be able to run commands inside that namespace with ip netns exec 1-de0d3f6935.

3. Gather information about the VXLAN interface

It seems that it is not possible to set the noudpcsum parameter on an already created interface so we must remove the VXLAN interface and recreate it. First we need to gather some information shown on the command:

ip netns exec 1-de0d3f6935 ip -d link show

Specifically we need the following:

1. The name of the interface [vxlan0]
2. The MTU [1424]
3. The VNI [4109]
4. The bridge to which the interface is enslaved to [br0]

4. Remove the VXLAN interface

Now that we have the necessary information we delete the interface with:

ip netns exec 1-de0d3f6935 ip link delete vxlan0

5. Recreate the VXLAN interface

An important point here is that the interface must be created in the default namespace because the socket on which the driver listens must be in that namespace. We will move the interface to the correct namespace on the next step.

ip link add vxlan0 mtu 1424 type vxlan vni 4109 dstport 4789 noudpcsum proxy l2miss l3miss

Note that we used the values that we gathered on step 3 and also that here we disable the UDP checksum. We must specify the port 4789 even though it is the default IANA port for VXLAN because Linux has a different default. Also note that we need the proxy, l2miss, and l3miss arguments because that is how the original interface was configured.

6. Move the VXLAN interface to the correct namespace

In this step we put the interface where it belongs. The socket listening on port 4789 will remain in the default namespace.

ip link set dev vxlan0 netns 1-de0d3f6935

7. Enslave the VXLAN interface to the bridge

The original interface was enslaved to a bridge so we do that with our new interface as well.

ip netns exec 1-de0d3f6935 ip link set dev vxlan0 master br0

8. Bring up the interface

ip netns exec 1-de0d3f6935 ip link set dev vxlan0 up

Once we do this in all the nodes everything will be ready but I found that I needed to restart the services which use the ingress network for things to work.

When creating an overlay network with encryption Docker should disable the UDP checksum on the VXLAN interfaces as integrity of the data is already guaranteed by ESP.

TL;DR: UDP checksum is wrong because ESP packets go through NAT. Recalculating the checksum solves the problem.

The overlay network is based on VXLAN, which works on top of UDP. When UDP datagrams traverse a NAT the source or destination IP address is changed, which means that the device providing the NAT must recalculate the UDP checksum. When the overlay network is configured to use encryption the VXLAN packets are protected with ESP in transport mode. In this case the NAT device can change the source or destination IP address but cannot update the UDP checksum. When these packets are decrypted they will have the original (i.e. before NAT) UDP checksum, which will be recognized as a bad checksum by the destination. The UDP checksum must be recalculated at the destination before the packets reach the VXLAN socket, otherwise they will be dropped. Another option, at least in theory, would be to somehow configure the VXLAN driver to not use the UDP checksum, that is, to always send the VXLAN packets with an UDP checksum of zero.

I was unable to correct the UDP checksum using iptables but I succeeded by using the tc-csum action. As far as I can tell, the tc actions must be used in conjunction with an interface. This means that I had to route the VXLAN packets through an interface onto which the tc-csum action was applied. I used a configuration similar to the following:

# Create a network namespace.
ip netns add csum

# Create two pairs of veth interfaces each with one member on 
# the default namespace and the other on the csum namespace.
ip link add csum_in type veth peer name in netns csum
ip link add csum_out type veth peer name out netns csum

# Assign IP addresses for the two pair of veth interfaces
# [csum_in, 192.168.117.1] <-> [in, 192.168.117.2]
# [csum_out, 192.168.118.2] <-> [out, 192.168.118.1]
# Remember that csum_in and csum_out belong to the default
# namespace while in and out beling to the csum namespace.

# Make it so that packets with bad checksums are considered
# for connection tracking.  This is necessary for the NAT rule
# that we'll configure next.
sysctl net.netfilter.nf_conntrack_checksum=0

# We want to route the VXLAN packets to a specific interface
# but they are addressed to this host, so we must change the
# destination address for now.  After we correct the checksum
# we'll restore the original destination address.  For this example
# lets say that the original IP address is 172.21.0.1 and the 
# temporary address is 172.31.0.1.
iptables -t nat -I PREROUTING 1 -d 172.21.0.1 -p udp --dport 4789 -j DNAT --to-destination 172.31.0.1
iptables -t nat -I PREROUTING 1 -d 172.31.0.1 -p udp --dport 4789 -j DNAT --to-destination 172.21.0.1

# We need to route the packets with the temporary address 
# so that they trigger the tc-csum action.
ip route add 172.31.0.1 via 192.168.117.2

# We need to be able to route the packets from the csum 
# namespace back to the default namespace.
ip netns exec csum ip route add 172.31.0.1 via 192.168.118.2

# If the FORWARD chain on the filter table has a DROP
# policy don't forget to ACCEPT the traffic on its way to
# the csum namespace.
iptables -t filter -A FORWARD -d 172.31.0.1 -p udp --dport 4789 -j ACCEPT

# Docker configures a rule in the INPUT chain of the filter
# table that DROPs VXLAN traffic that does not come via
# IPsec.  Either delete that rule or ACCEPT the traffic
# before it is dropped.
iptables -t filter -I INPUT <number> -d 172.21.0.1 -p udp --dport 4789 -j ACCEPT

# Be aware that some distributions have rp_filter=1.
# sysctl net.ipv4.conf.all.rp_filter=0
# sysctl net.ipv4.conf.csum_out.rp_filter=0
# ip netns exec csum sysctl net.ipv4.conf.all.rp_filter=0
# ip netns exec csum sysctl net.ipv4.conf.in.rp_filter=0

# This will recalculate the checksum of the traffic
# ingressing the csum namespace by the "in" interface.
ip netns exec csum tc qdisc add dev in ingress handle ffff:
ip netns exec csum tc filter add dev in prio 1 protocol ip parent ffff: u32 match ip src 172.31.0.1/32 flowid :1 action csum ip and udp

If you take a look at the iptables rules that Docker configures for the VXLAN traffic you’ll see that they match the VNI (by using the u32 match option). It would probably be a good idea to do this but I left that out for brevity. Note that even if you have NAT only on one node you will need to correct the checksum in all the nodes, because the NAT causes the checksum to be wrong on both ways.

@bijeebuss It cannot. IPSec NAT-T is not supported still.