tailscale: K8s node with tailscale transiently breaks dns inside containers

What is the issue?

see title 😦

I have a Linux machine running tailscale and acting as a k8s node/host. the kube distro is latest k3s.

example:

root@rbot-85fc96b69d-vkgbq:/app# while nslookup s3.us-east-1.amazonaws.com; do sleep 1; done
Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

Non-authoritative answer:
Name:   s3.us-east-1.amazonaws.com
Address: 52.217.111.102
Name:   s3.us-east-1.amazonaws.com
Address: 52.217.133.112
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.153.158
Name:   s3.us-east-1.amazonaws.com
Address: 54.231.131.72
Name:   s3.us-east-1.amazonaws.com
Address: 54.231.163.16
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.83.59
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.39.16
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.228.67

as you can see, it sometimes can’t reach amazon / chooses the wrong name to lookup? i’m not a dns expert tbh.

it’s not just aws s3 either, this happens with multiple domains.

this seems similar to https://github.com/tailscale/tailscale/issues/1003 but that’s marked as fixed and this clearly isnt.

note that beagle-chickadee.ts.net is my magicdns tailnet name, and i have magicdns enabled.

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Ubuntu 20.04

Tailscale version

on the host:

$ tailscale --version 1.32.3 tailscale commit: a07555f434843c2d049bfb04b0b17b61dd362824 other commit: 093d1e978538b249ab628c721e2f49d97b2cd16d go version: go1.19.2-ts3fd24dee31

Bug report

BUG-8da09730460803ece064ddc8b074cceff76d3adac875110f37a839d8eac46c36-20221207044352Z-a1bfef9102edc35e

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 2
  • Comments: 23 (8 by maintainers)

Most upvoted comments

Thank you @asg0451, I had a similar situation with my kubernetes cluster, headscale and tailscale-client w/magic dns and managed to resolve it with a minor change in your manifest:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-tailnet
  namespace: kube-system
data:
  <user>.<tailnet domain>.server: |
    <user>.<tailnet domain> {
      search .
    }
#Alternative solution as above did not work for Cilium CNI for some reason
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-tailnet
  namespace: kube-system
data:
  <user>.<tailnet domain>.server: |
    <user>.<tailnet domain> {
        rewrite stop {
          name regex (.*)[.](.*).<user>.<tailnet domain> {1}.{2}
          answer name (.*)[.](.*).<user>.<tailnet domain> {1}.{2}
        }
        forward . 100.100.100.100
    }
#this solution capitalizes on Golang regex and fact that magicdns rewrites every request to following format:
#nslookup google.com returns google.com.<user>.<tailnet domain>
#after applying above configmap, nslookup google.com returns google.com but preserves all requests that are not in X.Y format, allowing magicdns to function correctly for hostname lookups

with above applied, my pods can connect to internet without tls handshake errors.

Edit: added alternative solution. If your coredns does not accept custom configmaps as is the case with RKE2, above changes can be added to configmap/rke2-coredns-rke2-coredns in kube-system namespace.

based on the above, with some luck, i have a workaround:

# see https://blog.differentpla.net/blog/2022/02/25/coredns-custom/
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  beagle-chickadee.ts.net.server: |
    beagle-chickadee.ts.net {
      forward . 100.100.100.100
    }

this adds a new coredns ā€œserver blockā€ handling my tailnet. this works around the fact that coredns for Some Reason is using my network’s configured dns server instead of the node’s. this will break for other tailnets, but for me, for now, it should work

thanks for the help @DentonGentry, especially on a saturday night ā¤ļø

this is really hurting me… @bradfitz do you know of any remediation steps? this is the container’s resolv.conf btw:

search apps.svc.cluster.local svc.cluster.local cluster.local beagle-chickadee.ts.net
nameserver 10.43.0.10
options ndots:5

Got a tailscale bugreport?