tailscale: K8s node with tailscale transiently breaks dns inside containers

What is the issue?

see title 😦

I have a Linux machine running tailscale and acting as a k8s node/host. the kube distro is latest k3s.

example:

root@rbot-85fc96b69d-vkgbq:/app# while nslookup s3.us-east-1.amazonaws.com; do sleep 1; done
Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer

Server:         10.43.0.10
Address:        10.43.0.10#53

Non-authoritative answer:
Name:   s3.us-east-1.amazonaws.com
Address: 52.217.111.102
Name:   s3.us-east-1.amazonaws.com
Address: 52.217.133.112
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.153.158
Name:   s3.us-east-1.amazonaws.com
Address: 54.231.131.72
Name:   s3.us-east-1.amazonaws.com
Address: 54.231.163.16
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.83.59
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.39.16
Name:   s3.us-east-1.amazonaws.com
Address: 52.216.228.67

as you can see, it sometimes can’t reach amazon / chooses the wrong name to lookup? i’m not a dns expert tbh.

it’s not just aws s3 either, this happens with multiple domains.

this seems similar to https://github.com/tailscale/tailscale/issues/1003 but that’s marked as fixed and this clearly isnt.

note that beagle-chickadee.ts.net is my magicdns tailnet name, and i have magicdns enabled.

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Ubuntu 20.04

Tailscale version

on the host:

$ tailscale --version 1.32.3 tailscale commit: a07555f434843c2d049bfb04b0b17b61dd362824 other commit: 093d1e978538b249ab628c721e2f49d97b2cd16d go version: go1.19.2-ts3fd24dee31

Bug report

BUG-8da09730460803ece064ddc8b074cceff76d3adac875110f37a839d8eac46c36-20221207044352Z-a1bfef9102edc35e