tailscale: Tailscale >= 1.66.0 breaks DNS name resolution in Docker containers on Linux
What is the issue?
After upgrading to Tailscale 1.66.0 or 1.66.1, DNS name resolution breaks inside newly created Docker containers.
I’m experiencing this on a Debian 11 host.
This doesn’t occur on the same machine with Tailscale 1.64.0
Thanks for your support!
Steps to reproduce
- Install Tailscale 1.64.0 on the Linux machine:
apt install tailscale=1.64.0 - Start a Docker container:
docker run --rm -it debian:12 - Ping an external service from inside the container:
ping google.com-> WORKS - Stop the Docker containr:
exit - Install Tailscale 1.66.0 or 1.66.1 on the Linux machine:
apt install tailscale=1.66.1 - Start a Docker container:
docker run --rm -it debian:12 - Ping an external service from inside the container:
ping google.com-> Cannot resolve name
Are there any recent changes that introduced the issue?
Upgrade Tailscale from 1.64.0 to 1.66.1 using the system package manager.
OS
Linux
OS version
Debian 11
Tailscale version
1.66.1
Other software
Docker version 26.1.2, build 211e74b
Bug report
BUG-c7f85e39b9c72709e39b2f333de28e65a0d5875a2e351ab851dae9f245cea271-20240512175341Z-c93cbe3beb3e255b
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 24
- Comments: 32 (8 by maintainers)
Links to this issue
Commits related to this issue
- various: allowlist quad-100 DNS traffic from container interfaces Add a new prefs knob to allowlist DNS traffic from certain interfaces in the FORWARD chain (when source is not localhost). This is ma... — committed to tailscale/tailscale by awly 3 years ago
- various: allowlist quad-100 DNS traffic from container interfaces Add a new prefs knob to allowlist DNS traffic from certain interfaces in the FORWARD chain (when source is not localhost). This is ma... — committed to tailscale/tailscale by awly 3 years ago
- various: disable stateful filtering by default After some analysis, stateful filtering is only necessary in tailnets that use `autogroup:danger-all` in `src` in ACLs. And in those cases users explici... — committed to tailscale/tailscale by awly 3 years ago
- various: disable stateful filtering by default (#12197) After some analysis, stateful filtering is only necessary in tailnets that use `autogroup:danger-all` in `src` in ACLs. And in those cases us... — committed to tailscale/tailscale by awly 3 years ago
- various: disable stateful filtering by default (#12197) After some analysis, stateful filtering is only necessary in tailnets that use `autogroup:danger-all` in `src` in ACLs. And in those cases user... — committed to tailscale/tailscale by awly 3 years ago
- various: disable stateful filtering by default (#12197) After some analysis, stateful filtering is only necessary in tailnets that use `autogroup:danger-all` in `src` in ACLs. And in those cases us... — committed to Ckid-Home/tailscale by awly 3 years ago
- various: disable stateful filtering by default (#12197) After some analysis, stateful filtering is only necessary in tailnets that use `autogroup:danger-all` in `src` in ACLs. And in those cases us... — committed to Asutorufa/tailscale by awly 3 years ago
- cmd/k8s-operator: always set stateful filtering to false Updates tailscale/tailscale#12108 Signed-off-by: Irbe Krumina <irbe@tailscale.com> — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator: always set stateful filtering to false (#14216) Updates tailscale/tailscale#12108 Signed-off-by: Irbe Krumina <irbe@tailscale.com> — committed to tailscale/tailscale by irbekrm 2 years ago
- cmd/k8s-operator: always set stateful filtering to false (#14216) Updates tailscale/tailscale#12108 Signed-off-by: Irbe Krumina <irbe@tailscale.com> — committed to ninech/tailscale by irbekrm 2 years ago
I had the same problem which he solved:
tailscale set --stateful-filtering=falseYou owe me a good night’s sleep 😦
I had the same issue, which showed up as Service Widgets from my Homepage instance not connecting to their source (e.g. Proxmox, Jellyfin, etc).
The problem was solved by running
tailscale up --stateful-filtering=falseon my host machine.1.66 added stateful filtering for packets going via the
tailscale0interface, see https://tailscale.com/changelog#2024-05-08. But that should not affect packets from docker containers to the Internet.Another notable thing is that this node is also a subnet router for the
10.0.0.0/8range. Again, this should not affect docker containers, but noting it for others reading the issue.Some things to check are:
tailscale set --stateful-filtering=falseon 1.66.1 resolves the issue1.66.4 is available and disables stateful filtering by default. We discussed doing more clever things, like detecting container runtimes and allowlisting their interfaces in our iptables/nftables rules, but that gets pretty hairy. Instead, @maisem pointed out that other mitigations for https://tailscale.com/security-bulletins#ts-2024-005 cover most cases except for
autogroup:danger-all, so stateful filtering is not strictly required except for that case.If you run 1.66.0-1.66.3, you still need to manually disable stateful filtering with
tailscale set --stateful-filtering=false. If you install a brand new 1.66.4, or upgrade from 1.64 or older versions, stateful filtering will be off by default.Thanks for the details, I reproduced the issue. Right now,
tailscale set --stateful-filtering=falseis the only workaround. Running it on a regular node (not exit node and not subnet router) should be safe.The reason this happens is because Docker will by default use
resolv.conffrom the host, which points at 100.100.100.100 (served by local tailscaled) as the nameserver. Since this IP lands in thetailscale0interface, DNS request packets become subject to stateful filtering. And because the source IP is not localhost, but something like172.17.0.2(from the docker network range), it’s interpreted as “incoming request from another host” and thus rejected.While I don’t fully understand why yet, adding an additional resolve configuration to my
/etc/systemd/resolved.confseemingly fixes the issue:Setting the
--stateful-filtering=falsedoesn’t solve the issue for me. DNS still not working inside a docker container. I am using an AL2023+1
--stateful-filtering=falsesolve the problem@derpda the default change only applies to new installs. When you update from 1.66.3 (or older) to 1.66.4, the
--stateful-filteringvalue has already been set and does not revert to the new default. For existing installs that update to 1.66.4 runningtailscale set --stateful-filtering=falseis indeed the solution.@masterwishx just to clarify:
host.headscale.mysite.comworkshostdoes not workIf that’s correct, that sounds like a different issue. Can you show the contents of
/etc/resolv.confon the hosts that servers that don’t work, and those that do work?Dropping an updated command from what was posted above (https://github.com/tailscale/tailscale/issues/12108#issuecomment-2107156997) to view docker network names / IP ranges if you have jqlang/jq installed: