tailscale: FR: turn off routing DNS through exit node

What is the issue?

AFAIU all my DNS traffic should be direct between machines (using Exit Nodes and configured with Tailscale DNS settings) and the configured resolvers (100.X.X.X and 100.Y.Y.Y) in MagicDNS.

However, I am seeing an unexpected volume of DNS queries transiting through my Exit Nodes.

Steps to reproduce

On the Exit Node:

# tcpdump -i tailscale0 port 53 | grep pwet.com

Then on my machine configured with the Exit Node:

tcpdump output

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tailscale0, link-type RAW (Raw IP), capture size 262144 bytes
17:08:45.524647 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.41264 > 100.100.100.100.domain: 43223+ AAAA? thisisatestfromfirefox.pwet.com. (49)
17:08:45.525156 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.45975 > 100.100.100.100.domain: 47571+ A? thisisatestfromfirefox.pwet.com. (49)
17:08:45.525408 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.54917 > dns-euc1b-infra-02.company.com.beta.tailscale.net.domain: 43223+ AAAA? thisisatestfromfirefox.pwet.com. (49)
17:08:45.525475 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.36297 > dns-euc1b-infra-02.company.com.beta.tailscale.net.domain: 47571+ A? thisisatestfromfirefox.pwet.com. (49)
17:08:45.525554 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.38827 > dns-euc1a-infra-01.company.com.beta.tailscale.net.domain: 43223+ AAAA? thisisatestfromfirefox.pwet.com. (49)
17:08:45.525836 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.36497 > dns-euc1a-infra-01.company.com.beta.tailscale.net.domain: 47571+ A? thisisatestfromfirefox.pwet.com. (49)
17:08:59.378228 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.36736 > 100.100.100.100.domain: 60315+ A? thisisatestfrombrave.pwet.com. (47)
17:08:59.378439 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.52490 > dns-euc1b-infra-02.company.com.beta.tailscale.net.domain: 60315+ A? thisisatestfrombrave.pwet.com. (47)
17:08:59.378517 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.39328 > dns-euc1a-infra-01.company.com.beta.tailscale.net.domain: 60315+ A? thisisatestfrombrave.pwet.com. (47)
17:08:59.378866 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.39841 > 100.100.100.100.domain: 21670+ AAAA? thisisatestfrombrave.pwet.com. (47)
17:08:59.379038 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.58586 > dns-euc1b-infra-02.company.com.beta.tailscale.net.domain: 21670+ AAAA? thisisatestfrombrave.pwet.com. (47)
17:08:59.379436 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.52197 > dns-euc1a-infra-01.company.com.beta.tailscale.net.domain: 21670+ AAAA? thisisatestfrombrave.pwet.com. (47)
17:09:37.317700 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.48574 > 100.100.100.100.domain: 41354+ A? thisisatestfromclihost.pwet.com. (49)
17:09:37.317856 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.48980 > dns-euc1b-infra-02.company.com.beta.tailscale.net.domain: 41354+ A? thisisatestfromclihost.pwet.com. (49)
17:09:37.317939 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.38877 > dns-euc1a-infra-01.company.com.beta.tailscale.net.domain: 41354+ A? thisisatestfromclihost.pwet.com. (49)
17:09:37.440755 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.37163 > 100.100.100.100.domain: 48641+ AAAA? thisisatestfromclihost.pwet.com. (49)
17:09:37.440957 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.42811 > dns-euc1b-infra-02.company.com.beta.tailscale.net.domain: 48641+ AAAA? thisisatestfromclihost.pwet.com. (49)
17:09:37.441059 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.56174 > dns-euc1a-infra-01.company.com.beta.tailscale.net.domain: 48641+ AAAA? thisisatestfromclihost.pwet.com. (49)
17:09:37.652010 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.58013 > 100.100.100.100.domain: 43816+ MX? thisisatestfromclihost.pwet.com. (49)
17:09:37.652220 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.44638 > dns-euc1b-infra-02.company.com.beta.tailscale.net.domain: 43816+ MX? thisisatestfromclihost.pwet.com. (49)
17:09:37.652277 IP tailscale-euc1c-infra-03.company.com.beta.tailscale.net.38792 > dns-euc1a-infra-01.company.com.beta.tailscale.net.domain: 43816+ MX? thisisatestfromclihost.pwet.com. (49)

Only the resolution made with doggo is not routed through the exit node (as expected ?).

My /etc/resolv.conf

#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
search company.com.beta.tailscale.net
nameserver 100.100.100.100
$ scutil --dns
resolver #1
  search domain[0] : company.com.beta.tailscale.net
  nameserver[0] : 100.100.100.100
  if_index : 25 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102400

resolver #2
  nameserver[0] : 100.100.100.100
  if_index : 25 (utun3)
  flags    : Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 200000

resolver #3
  domain   : company.com.beta.tailscale.net.
  nameserver[0] : 100.100.100.100
  if_index : 25 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 102401

resolver #4
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #5
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #6
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #7
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #8
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #9
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 10.10.40.11
  nameserver[1] : 10.10.40.1
  if_index : 15 (en6)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  nameserver[0] : 10.10.40.11
  nameserver[1] : 10.10.198.1
  if_index : 14 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00000002 (Reachable)

resolver #3
  search domain[0] :company.com.beta.tailscale.net
  nameserver[0] : 100.100.100.100
  if_index : 25 (utun3)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)

Are there any recent changes that introduced the issue?

No response

OS

macOS

OS version

12.3 (21E230)

Tailscale version

1.22.1

Bug report

No response

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Comments: 16 (6 by maintainers)

Most upvoted comments

I came to this issue because as a user, it is surprising that DNS queries are forwarded to exit nodes, even when the DNS server is internal to the tailnet. However, with this new knowledge, I know that the exit node’s ACL needs to allow for it to reach the tailnet’s internal DNS server. So while surprising, the fix is trivial: add an ACL allowing the exit node to query the internal DNS server.

Perhaps this behavior violates the “principle of least surprise”, since from the user perspective, intra-tailnet traffic should be unaffected by whether or not one is using an exit node. I won’t argue that point one way or another, as the fix is simple, and reasonable people can disagree as to how “surprising” this behavior truly is. Just one additional perspective.

@DentonGentry

Not just “I want this” but what one expects to accomplish. The intent when using an exit node is to essentially teleport to the location of the exit node, and use its Internet connection as if one were there. This is the reason for forwarding DNS queries: DNS often gives different responses based on the geographic location of the client requesting it.

I think this behaviour is reasonable enough when considering a global DNS server (though it is still surprising), but it doesn’t make sense to defeat domain-specific overrides that have been explicitly configured in the control panel. Those rules most likely relate to split-DNS views or internal domains that are necessary for access to private services. I would consider custom DNS routes to be analogous to subnet routes, which are not overridden like DNS routes when enabling exit-node; they both relate to internal resources in one sense or another.

In my case I want to operate exit nodes in untrusted locations, without having to deploy a custom DNS resolver to those nodes with all my DNS policy duplicated, and without having to allow traffic back to the trusted part of the network from them. It makes exit node inconvenient for users since many internal resources are not accessible when it is enabled. Even if such a resolver is maintained on the exit node, it is likely suboptimal to route those queries via the exit node when the client already knows a direct path to the desired DNS server via the subnet router.

I’ve done a lot of searching and reading and I am fairly sure this is my issue.

I have a tailscale exit node and an Adguard DNS server (equivalent alternative to pihole) physically on the same box. When clients are out and about on a phone network or untrusted wifi, I want them to use both the exit node through the Tailscale tunnel and the Adguard DNS, but can’t. Only one or the other works.

On the server (Synology):

  • Tailscale 1.26.2 installed, advertising exit node as 100.83.xx.xx
  • Adguard DNS server installed for use by clients
  • Tailscale DNS global nameserver also set to 100.83.xx.xx, override local DNS on
  • MagicDNS disabled
  • DNS for the server’s own traffic set to 1.1.1.1

Client connection scenarios (various phones, laptops):

  • Connect to Tailscale using exit node: traffic is tunnelled but Adguard DNS is not used
  • Connect to Tailscale without exit node: Adguard DNS is used but traffic is not tunnelled

Now if I set the server to use the Adguard DNS for its own traffic too (i.e. change 1.1.1.1 to 127.0.0.1) then I can get the tunnelled exit node traffic to use the Adguard DNS. But as sylr commented above I hit this problem in that I can’t tell clients apart in the DNS audit logs because they are all proxied via the exit node and appear to all be coming from the one source.

One reason to do that is to still being able to audit the DNS queries made when using exit node (have the real client ips in pihole instead of the exit nodes ip).

Relevant: #3817

I am facing the same issue, can only use one or the other but not both on the same machine, the issue is still marked “Open”

@acaloiaro Can you tell me the steps for the fix: add an ACL allowing the exit node to query the internal DNS server. I tried finding the docs but was unable to find a clear instruction.

Assumptions:

  • Your DNS server is on port 53
  • Your exit node is tagged with exit-node

Add the following to your acls:

{
	"action": "accept",
	"src": ["tag:exit-node"],
	"dst": ["*:53"],
},

Note: This allows your exit node to connect to port 53 on any node in your tailnet. Change * to the IP of your DNS server for a stricter ACL.

I’ve done a lot of searching and reading and I am fairly sure this is my issue.

I have a tailscale exit node and an Adguard DNS server (equivalent alternative to pihole) physically on the same box. When clients are out and about on a phone network or untrusted wifi, I want them to use both the exit node through the Tailscale tunnel and the Adguard DNS, but can’t. Only one or the other works.

On the server (Synology):

  • Tailscale 1.26.2 installed, advertising exit node as 100.83.xx.xx
  • Adguard DNS server installed for use by clients
  • Tailscale DNS global nameserver also set to 100.83.xx.xx, override local DNS on
  • MagicDNS disabled
  • DNS for the server’s own traffic set to 1.1.1.1

Client connection scenarios (various phones, laptops):

  • Connect to Tailscale using exit node: traffic is tunnelled but Adguard DNS is not used
  • Connect to Tailscale without exit node: Adguard DNS is used but traffic is not tunnelled

Now if I set the server to use the Adguard DNS for its own traffic too (i.e. change 1.1.1.1 to 127.0.0.1) then I can get the tunnelled exit node traffic to use the Adguard DNS. But as sylr commented above I hit this problem in that I can’t tell clients apart in the DNS audit logs because they are all proxied via the exit node and appear to all be coming from the one source.

One reason to do that is to still being able to audit the DNS queries made when using exit node (have the real client ips in pihole instead of the exit nodes ip).

Relevant: https://github.com/tailscale/tailscale/issues/3817