tailscale: K8s node with tailscale transiently breaks dns inside containers
What is the issue?
see title š¦
I have a Linux machine running tailscale and acting as a k8s node/host. the kube distro is latest k3s.
example:
root@rbot-85fc96b69d-vkgbq:/app# while nslookup s3.us-east-1.amazonaws.com; do sleep 1; done
Server: 10.43.0.10
Address: 10.43.0.10#53
*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer
Server: 10.43.0.10
Address: 10.43.0.10#53
*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer
Server: 10.43.0.10
Address: 10.43.0.10#53
*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer
Server: 10.43.0.10
Address: 10.43.0.10#53
*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer
Server: 10.43.0.10
Address: 10.43.0.10#53
*** Can't find s3.us-east-1.amazonaws.com.beagle-chickadee.ts.net: No answer
Server: 10.43.0.10
Address: 10.43.0.10#53
Non-authoritative answer:
Name: s3.us-east-1.amazonaws.com
Address: 52.217.111.102
Name: s3.us-east-1.amazonaws.com
Address: 52.217.133.112
Name: s3.us-east-1.amazonaws.com
Address: 52.216.153.158
Name: s3.us-east-1.amazonaws.com
Address: 54.231.131.72
Name: s3.us-east-1.amazonaws.com
Address: 54.231.163.16
Name: s3.us-east-1.amazonaws.com
Address: 52.216.83.59
Name: s3.us-east-1.amazonaws.com
Address: 52.216.39.16
Name: s3.us-east-1.amazonaws.com
Address: 52.216.228.67
as you can see, it sometimes canāt reach amazon / chooses the wrong name to lookup? iām not a dns expert tbh.
itās not just aws s3 either, this happens with multiple domains.
this seems similar to https://github.com/tailscale/tailscale/issues/1003 but thatās marked as fixed and this clearly isnt.
note that beagle-chickadee.ts.net is my magicdns tailnet name, and i have magicdns enabled.
Steps to reproduce
No response
Are there any recent changes that introduced the issue?
No response
OS
Linux
OS version
Ubuntu 20.04
Tailscale version
on the host:
$ tailscale --version 1.32.3 tailscale commit: a07555f434843c2d049bfb04b0b17b61dd362824 other commit: 093d1e978538b249ab628c721e2f49d97b2cd16d go version: go1.19.2-ts3fd24dee31
Bug report
BUG-8da09730460803ece064ddc8b074cceff76d3adac875110f37a839d8eac46c36-20221207044352Z-a1bfef9102edc35e
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 2
- Comments: 23 (8 by maintainers)
Thank you @asg0451, I had a similar situation with my kubernetes cluster, headscale and tailscale-client w/magic dns and managed to resolve it with a minor change in your manifest:
with above applied, my pods can connect to internet without tls handshake errors.
Edit: added alternative solution. If your coredns does not accept custom configmaps as is the case with RKE2, above changes can be added to configmap/rke2-coredns-rke2-coredns in kube-system namespace.
based on the above, with some luck, i have a workaround:
this adds a new coredns āserver blockā handling my tailnet. this works around the fact that coredns for Some Reason is using my networkās configured dns server instead of the nodeās. this will break for other tailnets, but for me, for now, it should work
thanks for the help @DentonGentry, especially on a saturday night ā¤ļø
this is really hurting me⦠@bradfitz do you know of any remediation steps? this is the containerās resolv.conf btw:
Got a
tailscale bugreport?