tailscale: Tailscaled fails to start when IPv6 policy routing is unavailable

Hi,

Description

Since updating tailscale to version 1.2.1, it seems it does not create the socket in /var/run/tailscale/tailscaled.sock. The systemd service is indeed started but tailscale itself cannot connect or be brought up.

What happens

Tailscale service is running but /usr/bin/tailscale cannot connect

What should happen

Update should not break existing configuration

Steps to reproduce

Update tailscale through sudo apt update; sudo apt upgrade

Technical details

OS - Debian stable :

<font color="#E9AD0C">admin</font>@lovelace:<font color="#2AA1B3">[~]</font>: cat /etc/debian_version 
10.6

Tailscale Version:

<font color="#E9AD0C">admin</font>@lovelace:<font color="#2AA1B3">[~]</font>: tailscale --version
1.2.1
  tailscale commit: 9446e5c170fb251b83d3e1d14b319991c32f4135
  other commit: 351e0990591be7c5f6631cca09c68ac264671404-dirty
  go version: go1.15-tsc333f7a

Kernel Version:

<font color="#E9AD0C">admin</font>@lovelace:<font color="#2AA1B3">[~]</font>: uname -a 
Linux lovelace 4.4.114-mainline-rev1 #1 SMP Thu Feb 1 16:16:01 UTC 2018 x86_64 GNU/Linux

Systemd service status:

<font color="#E9AD0C">admin</font>@lovelace:<font color="#2AA1B3">[~]</font>: sudo systemctl status tailscaled.service 
<font color="#26A269">●</font> tailscaled.service - Tailscale node agent
   Loaded: loaded (/lib/systemd/system/tailscaled.service; enabled; vendor preset: enabled)
   Active: <font color="#26A269">active (running)</font> since Wed 2020-11-04 16:48:24 UTC; 933ms ago
     Docs: https://tailscale.com/kb/
 Main PID: 27057 (tailscaled)
    Tasks: 9 (limit: 2389)
   Memory: 9.6M
   CGroup: /system.slice/tailscaled.service
           └─27057 /usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port 41641

Nov 04 16:48:24 lovelace tailscaled[27057]: router: dns: using dns.resolvconfManager
Nov 04 16:48:24 lovelace tailscaled[27057]: Bringing wireguard device up...
Nov 04 16:48:24 lovelace tailscaled[27057]: Bringing router up...
Nov 04 16:48:24 lovelace tailscaled[27057]: Device closing
Nov 04 16:48:24 lovelace tailscaled[27057]: Routine: event worker - stopped
Nov 04 16:48:24 lovelace tailscaled[27057]: [RATE LIMITED] %s
Nov 04 16:48:24 lovelace tailscaled[27057]: wgengine.New: running "ip -6 rule add pref 5210 fwmark 0x80000 table main" failed: exit status 2
Nov 04 16:48:24 lovelace tailscaled[27057]: RTNETLINK answers: Address family not supported by protocol
Nov 04 16:48:24 lovelace tailscaled[27057]: flushing log.
Nov 04 16:48:24 lovelace tailscaled[27057]: logger closing down

tailscale error:

<font color="#E9AD0C">admin</font>@lovelace:<font color="#2AA1B3">[~]</font>: tailscale up
Failed to connect to connect to tailscaled. (safesocket.Connect: dial unix /var/run/tailscale/tailscaled.sock: connect: no such file or directory)

Mitigations tried

I tried to reinstall, purge and reinstall to no avail.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 29 (11 by maintainers)

Commits related to this issue

Most upvoted comments

Hi, I have a slightly similar error upon launching Tailscale.

Following the steps above, I am running Tailscale (formerly on 1.2.2, then upgraded to 1.3.46 following this thread), on an Ubuntu 18.04 machine that has to deal with a HTTP proxy (it is configured, I am able to curl http and https). IPv6 is disabled.

error

nvidia-lmr@nvidia-lmr:~$ sudo tailscale up 2020/11/13 16:20:08 ReadMsg: read unix @->/run/tailscale/tailscaled.sock: use of closed network connection

Status of the service :

nvidia-lmr@nvidia-lmr:~$ sudo systemctl status tailscaled.service
tailscaled.service - Tailscale node agent
   Loaded: loaded (/lib/systemd/system/tailscaled.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-11-13 16:19:47 CET; 59s ago
     Docs: https://tailscale.com/kb/
  Process: 28941 ExecStopPost=/usr/sbin/tailscaled --cleanup (code=exited, status=0/SUCCESS)
  Process: 28954 ExecStartPre=/usr/sbin/tailscaled --cleanup (code=exited, status=0/SUCCESS)
 Main PID: 28994 (tailscaled)
    Tasks: 15 (limit: 4915)
   CGroup: /system.slice/tailscaled.service
           └─28994 /usr/sbin/tailscaled --state=/var/lib/tailscale/tailscaled.state --socket=/run/tailscale/tailscaled.sock --port 41641

nov. 13 16:19:51 nvidia-lmr tailscaled[28994]: control: client.Login(false, 0)
nov. 13 16:19:51 nvidia-lmr tailscaled[28994]: control: authRoutine: context done.
nov. 13 16:19:51 nvidia-lmr tailscaled[28994]: control: authRoutine: state:new
nov. 13 16:19:51 nvidia-lmr tailscaled[28994]: control: direct.TryLogin(token=false, flags=0)
nov. 13 16:19:51 nvidia-lmr tailscaled[28994]: control: doLogin(regen=false, hasUrl=false)
nov. 13 16:20:08 nvidia-lmr tailscaled[28994]: ipnserver: conn1: ReadMsg: EOF
nov. 13 16:20:10 nvidia-lmr tailscaled[28994]: magicsock: periodicReSTUN disabled due to inactivity
nov. 13 16:20:18 nvidia-lmr tailscaled[28994]: logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: i/o timeout (in 30.005s)
nov. 13 16:20:18 nvidia-lmr tailscaled[28994]: logtail: upload: log upload of 24967 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/a05828fc3c2d8bd65c7050258d8a40d40299edda76bd155dd2283e7f7d39e47d": dial tcp 34.210.105.16:443: i/o timeo
nov. 13 16:20:18 nvidia-lmr tailscaled[28994]: logtail: backoff: 5 msec

`

And the journal state after sudo tailscale up :

nvidia-lmr@nvidia-lmr:~$ sudo journalctl -u tailscaled --since="2 minutes ago"
-- Logs begin at Sun 2018-01-28 16:58:17 CET, end at Fri 2020-11-13 16:21:55 CET. --
nov. 13 16:20:08 nvidia-lmr tailscaled[28994]: ipnserver: conn1: ReadMsg: EOF
nov. 13 16:20:10 nvidia-lmr tailscaled[28994]: magicsock: periodicReSTUN disabled due to inactivity
nov. 13 16:20:18 nvidia-lmr tailscaled[28994]: logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: i/o timeout (in 30.005s)
nov. 13 16:20:18 nvidia-lmr tailscaled[28994]: logtail: upload: log upload of 24967 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/a05828fc3c2d8bd65c7050258d8a40d40299edda76bd155dd2283e7f7d39e47d": dial tcp 34.210.105.16:443: i/o timeo
nov. 13 16:20:18 nvidia-lmr tailscaled[28994]: logtail: backoff: 5 msec
nov. 13 16:20:48 nvidia-lmr tailscaled[28994]: logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: i/o timeout (in 30.004s)
nov. 13 16:20:48 nvidia-lmr tailscaled[28994]: logtail: upload: log upload of 24967 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/a05828fc3c2d8bd65c7050258d8a40d40299edda76bd155dd2283e7f7d39e47d": dial tcp 34.210.105.16:443: i/o timeo
nov. 13 16:20:48 nvidia-lmr tailscaled[28994]: logtail: backoff: 42 msec
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: ipnserver: conn2: incoming control connection
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: SetPrefs: Prefs{ra=false dns=true want=true routes=[] nf=on Persist=nil}
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: netmap packet filter: (not ready yet)
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: authReconfig: netmap not yet valid. Skipping.
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: Start
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: client.Shutdown()
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: client.Shutdown: inSendStatus=0
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: TryLogin: fetch control key: Get "https://login.tailscale.com/key": context canceled
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: authRoutine: state:authenticating
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: authRoutine: quit
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: mapRoutine: context done.
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: mapRoutine: state:authenticating
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: mapRoutine: quit
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: Client.Shutdown done.
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: using backend prefs
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: backend prefs for "_daemon": Prefs{ra=false dns=true want=true routes=[] nf=on Persist=nil}
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: netmap packet filter: (not ready yet)
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: HostInfo: {"IPNVersion":"1.3.46-ta664aac87-g3c383a153","BackendLogID":"815ca8653ef11fc4c69b71109a0394ee725979ec444691d27c390f010748d846","OS":"linux","OSVersion":"Ubuntu 18.04.2 LTS (Bionic Beaver); kernel=4.9.140-te
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: Backend: logs: be:815ca8653ef11fc4c69b71109a0394ee725979ec444691d27c390f010748d846 fe:
nov. 13 16:21:16 nvidia-lmr tailscaled[28994]: control: client.Login(false, 0)

The weird thing, is that I am able to curl https://login.tailscale.com/key from the cmd line.

Tailscale netcheck

nvidia-lmr@nvidia-lmr:~$ tailscale netcheck

Report:
	* UDP: false
	* IPv4: (no addr found)
	* IPv6: no
	* MappingVariesByDestIP:
	* HairPinning:
	* PortMapping:
	* Nearest DERP: unknown (no response to latency probes)

I am investigating my proxy configuration, but as internet connection is working fine (curl, apt-get …) apart from Tailscale, I am not sure what I could try now.

+1
barth-azmed on Nov 13, 2020

Released 1.2.6 with this fix. You should be fine to switch back to the stable track now (once you’ve switched back to the stable repo URL, apt-get install tailscale=1.2.6 should switch you over, if I remember my incantations correctly)

+1
danderson on Nov 13, 2020

Yeah, you have a different problem, which is that the tuntap device driver isn’t loaded. Based on:

Nov 12 11:34:39 odroidhc1 tailscaled[31671]: Linux kernel version: 4.14.187-odroidxu4
Nov 12 11:34:39 odroidhc1 tailscaled[31671]: is CONFIG_TUN enabled in your kernel? modprobe tun failed with: mo
Nov 12 11:34:39 odroidhc1 tailscaled[31671]: modprobe: FATAL: Module tun not found in directory /lib/modules/4.14
Nov 12 11:34:40 odroidhc1 tailscaled[31671]: kernel/drivers/net/tun.ko found on disk, but not for current kernel;
Nov 12 11:34:40 odroidhc1 tailscaled[31671]: CreateTUN: CreateTUN("tailscale0") failed; /dev/net/tun does not exist

you may need to reboot, because there is a tuntap driver on disk, but for the wrong kernel version. Usually that means the kernel package got updated, but you haven’t rebooted yet so the modules don’t match.

+1
danderson on Nov 12, 2020

I just pushed unstable packages to https://pkgs.tailscale.com/unstable. Can y’all give that a try and see if it fixes things?

+1
danderson on Nov 11, 2020
Read more comments on GitHub
← tailscale: tailscaled doesn't start on Google Colaboratory
tailscale: Timeout with `ping` but not with `tailscale ping`; suspect local firewall →