tailscale: Tailscale >= 1.66.0 breaks DNS name resolution in Docker containers on Linux

What is the issue?

After upgrading to Tailscale 1.66.0 or 1.66.1, DNS name resolution breaks inside newly created Docker containers.

I’m experiencing this on a Debian 11 host.

This doesn’t occur on the same machine with Tailscale 1.64.0

Thanks for your support!

Steps to reproduce

  1. Install Tailscale 1.64.0 on the Linux machine: apt install tailscale=1.64.0
  2. Start a Docker container: docker run --rm -it debian:12
  3. Ping an external service from inside the container: ping google.com -> WORKS
  4. Stop the Docker containr: exit
  5. Install Tailscale 1.66.0 or 1.66.1 on the Linux machine: apt install tailscale=1.66.1
  6. Start a Docker container: docker run --rm -it debian:12
  7. Ping an external service from inside the container: ping google.com -> Cannot resolve name

Are there any recent changes that introduced the issue?

Upgrade Tailscale from 1.64.0 to 1.66.1 using the system package manager.

OS

Linux

OS version

Debian 11

Tailscale version

1.66.1

Other software

Docker version 26.1.2, build 211e74b

Bug report

BUG-c7f85e39b9c72709e39b2f333de28e65a0d5875a2e351ab851dae9f245cea271-20240512175341Z-c93cbe3beb3e255b

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 24
  • Comments: 32 (8 by maintainers)

Commits related to this issue

Most upvoted comments

I had the same problem which he solved: tailscale set --stateful-filtering=false

You owe me a good night’s sleep 😦

I had the same issue, which showed up as Service Widgets from my Homepage instance not connecting to their source (e.g. Proxmox, Jellyfin, etc).

The problem was solved by running tailscale up --stateful-filtering=false on my host machine.

1.66 added stateful filtering for packets going via the tailscale0 interface, see https://tailscale.com/changelog#2024-05-08. But that should not affect packets from docker containers to the Internet.

Another notable thing is that this node is also a subnet router for the 10.0.0.0/8 range. Again, this should not affect docker containers, but noting it for others reading the issue.

Some things to check are:

  • check whether tailscale set --stateful-filtering=false on 1.66.1 resolves the issue
  • check if your docker configuration uses CGNAT IPs (in the 100.64.0.0/10 range) by chance? You can run something like this to view the config for all docker networks:
    docker network ls --format '{{.Name}}' | xargs docker network inspect
    

1.66.4 is available and disables stateful filtering by default. We discussed doing more clever things, like detecting container runtimes and allowlisting their interfaces in our iptables/nftables rules, but that gets pretty hairy. Instead, @maisem pointed out that other mitigations for https://tailscale.com/security-bulletins#ts-2024-005 cover most cases except for autogroup:danger-all, so stateful filtering is not strictly required except for that case.

If you run 1.66.0-1.66.3, you still need to manually disable stateful filtering with tailscale set --stateful-filtering=false. If you install a brand new 1.66.4, or upgrade from 1.64 or older versions, stateful filtering will be off by default.

Thanks for the details, I reproduced the issue. Right now, tailscale set --stateful-filtering=false is the only workaround. Running it on a regular node (not exit node and not subnet router) should be safe.

The reason this happens is because Docker will by default use resolv.conf from the host, which points at 100.100.100.100 (served by local tailscaled) as the nameserver. Since this IP lands in the tailscale0 interface, DNS request packets become subject to stateful filtering. And because the source IP is not localhost, but something like 172.17.0.2 (from the docker network range), it’s interpreted as “incoming request from another host” and thus rejected.

While I don’t fully understand why yet, adding an additional resolve configuration to my /etc/systemd/resolved.conf seemingly fixes the issue:

[Resolve]
DNS=100.100.100.100
Domains=your-tailnet-name.ts.net

Setting the --stateful-filtering=false doesn’t solve the issue for me. DNS still not working inside a docker container. I am using an AL2023

+1

--stateful-filtering=false solve the problem

@derpda the default change only applies to new installs. When you update from 1.66.3 (or older) to 1.66.4, the --stateful-filtering value has already been set and does not revert to the new default. For existing installs that update to 1.66.4 running tailscale set --stateful-filtering=false is indeed the solution.

@masterwishx just to clarify:

  • resolving full names, like host.headscale.mysite.com works
  • resolving short names, like host does not work

If that’s correct, that sounds like a different issue. Can you show the contents of /etc/resolv.conf on the hosts that servers that don’t work, and those that do work?

Dropping an updated command from what was posted above (https://github.com/tailscale/tailscale/issues/12108#issuecomment-2107156997) to view docker network names / IP ranges if you have jqlang/jq installed:

docker network ls --format '{{.Name}}' | xargs docker network inspect | jq '.[] | [.Name, .IPAM.Config]'

Cursor_and_zsh