tailscale: tailscaled doesn't start on Google Colaboratory

Describe the bug I tried to use tailscale on Google Colaboratory but I got error when I tried to run tailscaled --state=tailscaled.state.

System info

!cat /etc/lsb-release
!uname -a

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
Linux  4.19.104+ #1 SMP Wed Feb 19 05:26:34 PST 2020 x86_64 x86_64 x86_64 GNU/Linux

CONFIG_TUN is available:

!zcat /proc/config.gz | grep CONFIG_TUN

CONFIG_TUN=m
# CONFIG_TUN_VNET_CROSS_LE is not set

Container info using container introspection tool: https://github.com/genuinetools/amicontained

!amicontained

Container Runtime: docker
Has Namespaces:
	pid: true
	user: false
AppArmor Profile: datalabvm (enforce)
Capabilities:
	BOUNDING -> chown dac_override fowner fsetid kill setgid setuid setpcap net_bind_service net_raw sys_chroot sys_ptrace sys_admin mknod audit_write setfcap
Seccomp: filtering
Blocked Syscalls (47):
	SETSID USELIB USTAT SYSFS VHANGUP PIVOT_ROOT _SYSCTL ACCT SETTIMEOFDAY SWAPON SWAPOFF REBOOT IOPL IOPERM CREATE_MODULE INIT_MODULE DELETE_MODULE GET_KERNEL_SYMS QUERY_MODULE NFSSERVCTL GETPMSG PUTPMSG AFS_SYSCALL TUXCALL SECURITY CLOCK_SETTIME VSERVER MBIND SET_MEMPOLICY GET_MEMPOLICY KEXEC_LOAD ADD_KEY REQUEST_KEY KEYCTL MIGRATE_PAGES MOVE_PAGES OPEN_BY_HANDLE_AT CLOCK_ADJTIME FINIT_MODULE KEXEC_FILE_LOAD USERFAULTFD MEMBARRIER PKEY_MPROTECT PKEY_ALLOC PKEY_FREE IO_PGETEVENTS RSEQ

To Reproduce I installed tailscale with following command:

!curl https://pkgs.tailscale.com/stable/ubuntu/bionic.gpg | sudo apt-key add -
!curl https://pkgs.tailscale.com/stable/ubuntu/bionic.list | sudo tee /etc/apt/sources.list.d/tailscale.list
!sudo apt-get update
!sudo apt-get install tailscale

As systemd is not working, tailscaled need to be run manually but failed:

!tailscaled --state=tailscaled.state

2020/08/03 00:33:30 logpolicy.Read /root/.cache/Tailscale/tailscaled.log.conf: open /root/.cache/Tailscale/tailscaled.log.conf: no such file or directory
logtail started
Program starting: v0.100.0-153-gc6c838785, Go 1.14.4-ts56db765: []string{"tailscaled", "--state=tailscaled.state"}
LogID: 3f96acd67e70b830a5a2b0482eb3e276917c58fc9774164c4c7cd1e3f83250a5
6.4M/93.0M Starting userspace wireguard engine with tun device "tailscale0"
3.1M/93.0M Linux kernel version: 4.19.104+
3.1M/93.0M is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/4.19.104+
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 59ms)
logtail: upload: log upload of 252 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 12 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 252 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 58 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 252 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 121 msec
3.1M/93.0M tun module not loaded nor found on disk
3.1M/93.0M CreateTUN: can't create TUN device; /dev/net/tun does not exist
wgengine.New: can't create TUN device; /dev/net/tun does not exist

I read https://github.com/tailscale/tailscale/issues/504 and created /dev/net/tun:

!mkdir /dev/net
!mknod /dev/net/tun c 10 200
!chmod 0666 /dev/net/tun

Then I tried to run tailscaled again: (modprobe tun failed because Google colab runtime uses kernel 4.19 but there is no /lib/modules/4.19.104 directory but /lib/modules/4.15.0-112-generic)

!tailscaled --state=tailscaled.state

logtail started
Program starting: v0.100.0-153-gc6c838785, Go 1.14.4-ts56db765: []string{"tailscaled", "--state=tailscaled.state"}
LogID: 3f96acd67e70b830a5a2b0482eb3e276917c58fc9774164c4c7cd1e3f83250a5
5.2M/93.0M Starting userspace wireguard engine with tun device "tailscale0"
3.1M/93.0M Linux kernel version: 4.19.104+
3.1M/93.0M is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with: modprobe: FATAL: Module tun not found in directory /lib/modules/4.19.104+
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 56ms)
logtail: upload: log upload of 756 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 12 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 756 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 58 msec
logtail: dial "log.tailscale.io:443" failed: dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted (in 2ms)
logtail: upload: log upload of 756 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/005c0b9d5b8c3fe8d59022f7357148dad300899f9762ccfaa53bf2e989dc8365": dial tcp 34.210.105.16:443: setting SO_MARK bypass: operation not permitted
logtail: backoff: 121 msec
3.3M/93.0M tun module not loaded nor found on disk
3.3M/93.0M CreateTUN: operation not permitted
wgengine.New: operation not permitted

That error message says tun module not loaded but there is tun module:

!lsmod

Module                  Size  Used by
tun                    49152  0
xt_nat                 16384  8
veth                   24576  0
cls_u32                24576  2
sch_htb                24576  1
ipt_MASQUERADE         16384  1
iptable_nat            16384  3
nf_nat_ipv4            16384  2 ipt_MASQUERADE,iptable_nat
xt_addrtype            16384  2
nf_nat                 53248  2 nf_nat_ipv4,xt_nat
br_netfilter           24576  0
ip6table_filter        16384  1
ip6_tables             28672  1 ip6table_filter
aesni_intel           200704  0
virtio_balloon         20480  0
aes_x86_64             20480  1 aesni_intel
crypto_simd            16384  1 aesni_intel
cryptd                 28672  2 crypto_simd,aesni_intel
glue_helper            16384  1 aesni_intel

This document says CAP_NET_ADMIN is required for creating network devices. https://github.com/torvalds/linux/blob/master/Documentation/networking/tuntap.rst

It seems tailscaled failed to run because google colab runtime doesn’t have CAP_NET_ADMIN capability. tailscale cannot work without CAP_NET_ADMIN capability?

I have a python script that automatically setup ssh server on Google Colaboratory and allow login to it using ngrok. https://github.com/demotomohiro/remocolab And I tried to use tailscale on Google Colaboratory because tailscale looks faster and more secure alternative to ngrok.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (14 by maintainers)

Commits related to this issue

Most upvoted comments

The code which calls SO_MARK has:

func ignoreErrors() bool {
        // If we're in a test, ignore errors. Assume the test knows
        // what it's doing and will do its own skips or permission
        // checks if it's setting up a world that needs netns to work.
        // But by default, assume that tests don't need netns and it's
        // harmless to ignore the sockopts failing.
        if flag.CommandLine.Lookup("test.v") != nil {
                return true
        }
        if os.Getuid() != 0 {
                // only root can manipulate these socket flags
                return true
        }
        return false
}

This is either too clever or not clever enough: Colab runs the binary as root, but SO_MARK isn’t allowed. So we exit tailscaled.

We can get around this even in the current binary by running it as something non-root. I chose IRC because, as everyone knows, IRC is all powerful. I was able to get a Colab notebook to join the Tailnet like so:

!curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.gpg | sudo apt-key add -
!curl -fsSL https://pkgs.tailscale.com/stable/ubuntu/bionic.list | sudo tee /etc/apt/sources.list.d/tailscale.list
!sudo apt-get update
!sudo apt-get install tailscale
!rm -rf /tmp/tailscaled
!mkdir -p /tmp/tailscaled
!chown irc.irc /tmp/tailscaled
!rm -rf /var/run/tailscale
!mkdir -p /var/run/tailscale
!chown irc.irc /var/run/tailscale
!cp /var/lib/tailscaled/tailscaled.state /tmp/tailscaled/tailscaled.state
!chown irc.irc /tmp/tailscaled/tailscaled.state
!nohup sudo -u irc tailscaled --tun=userspace-networking --socks5-server=localhost:1055 --state=/tmp/tailscaled/tailscaled.state --socket=/var/run/tailscale/tailscaled.sock --port 41641 &
!until tailscale up; do sleep 1; done

It printed a message:

To authenticate, visit:

	https://login.tailscale.com/a/a-long-code

which I clicked on to authorize it. I now have a machine named 79c8ee43a59f in my Tailnet, which I immediately renamed as “George.” As one does.

I’ll work on making this nicer by ignoring a SO_MARK failure in Colab, even when running as root. This will remove the need for most of that ugly scripting as it will be able run with its normal paths.

To set expectations: running userspace-networking as a SOCKS5 server means that regular shell commands run from Colab !script will not suddenly be able to reach addresses on the Tailnet. Binaries have to be told to connect to the SOCKS5 proxy being run by tailscaled, for example:

!curl --socks5-hostname localhost:1055 http://100.100.1.1/some/path/to/something/

+2
DentonGentry on Apr 9, 2021

Once we get into hosting an http proxy inside tailscaled, there are actually tons of use cases for this that would work well, similarly to sshuttle. Another great use case would be running as non-admin on Windows machines. ᐧ

On Mon, Aug 3, 2020 at 4:57 PM Dave Anderson notifications@github.com wrote:

Such a tailscaled would have limited functionality compared to a regular system. It’s interesting to explore at some point, assuming this environment can’t be made more friendly, but it’s not a small undertaking.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tailscale/tailscale/issues/634#issuecomment-668237620, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAFA4EMRAI2IF64NIMPVB3R64QB3ANCNFSM4PTWAZ7A .

– Avery Pennarun // CEO @ Tailscale

+1
apenwarr on Aug 4, 2020
Read more comments on GitHub
← tailscale: tailscaled 1.36.2 on macOS Ventura does not start on machine boot
tailscale: Tailscaled fails to start when IPv6 policy routing is unavailable →