moby: Cannot receive UDP traffic in Docker container in case of UDP server being located at Docker host

same here with docker v1.9.1 (“a34a1d5-dirty”) @ Arch

👍 please fix this bug

+1

The problem happens because host has multiple network interfaces(eth0 and docker0 in this case), and when replying UDP, server choose the wrong IP as the source address.

Running tcpdump to capture packets on the wire gives:

14:42:11.861765 IP 172.17.0.3.37218 > 172.16.13.13.5151: UDP, length 6
14:42:11.863471 IP 172.17.0.1.5151 > 172.17.0.3.37218: UDP, length 6
14:42:11.863520 IP 172.17.0.3 > 172.17.0.1: ICMP 172.17.0.3 udp port 37218 unreachable, length 42

• The first packet means client sends a UDP request to server, with source address 172.17.0.3(container IP address), and destination address 172.16.13.13(host eth0 address)
• The second packets is the response, however, source address becomes 172.17.0.1(docker0 ip address), instead of 172.16.13.13 as we expected
• The third packet is sent by client, telling server that the second packet is INVALID

As we can see the real cause is: the second packet chooses the wrong source address! And somehow, client does not recognize it, thinks it is INVALID, does not pass it to the application, but instead replies with a ICMP packet.

Choosing packet source address is kernel’s work, for UDP, by default it will use the IP address of the interface the packet will be sent to. And as to why kernel drops the second packet, is because netfilter records valid UDP connection(since UDP is connectionless, it only means netfilter stores <sourcess IP, source Port, destination IP, destination Port>), and drops those that are invalid(not in connection table).

As for the first problem, socket can actually control which source address should be used with a combination of sendmsg and IP_PKTINFO socket option. See how powerDNS does it: https://blog.powerdns.com/2012/10/08/on-binding-datagram-udp-sockets-to-the-any-addresses/

In conclusion, this is not a problem caused by docker(although it does create a docker0 which is the beginning of it) . And the solution comes in many ways:

• use TCP instead of UDP, because TCP is connected protocol, kernel will use the right address
• server should listen on a specific IP address
• Change server socket logic, please refer to: https://github.com/netoptimizer/network-testing/blob/master/src/udp_example02.c

1 month later and still no progress

Unbelievable, bug still in current docker version and no one cares from docker dev team (@thaJeztah).

I got an issue with UDP packets and a docker-container. In my case, I solved that in my docker-compose file adding “/udp” in the ports section.

ports:
   "5223:5223/udp"

Source: https://docs.docker.com/config/containers/container-networking/

Also hitting this issue… any fix for this ?

running osx Sierra too

@TimothyKlim sorry about that. this fell out of radar. We will take a look at it soon.

+1

I’m having a [possibly] related issue.

My app receives UDP syslog messages from most sources with the original source address preserved, but one device has the source rewritten to the docker0 interface’s IP (seen by running tcpdump on docker0).

I tried disabling the userland proxy as an experiment. The ones that were working before still are, but that one source never makes it to docker0 now, which is obviously concerning.

Ubuntu 16.04.2, docker 1.12.6, docker-compose 1.8.0, launched w/ docker-compose up

EDIT: Running conntrack -D -p udp seems to have fixed it for me locally, so potentially unrelated.

Apparently my issue was I needed to bind dnsmasq to a specific IP address not 0.0.0.0 as per https://blog.powerdns.com/2012/10/08/on-binding-datagram-udp-sockets-to-the-any-addresses/

+1