tailscale: tailscaled 1.36.2 on macOS Ventura does not start on machine boot

What is the issue?

We recently upgraded some of our managed computers to macOS Ventura from macOS Monterey. We have been using Tailscale successfully on these computers for many months and not faced any issues with Tailscale starting on boot while on macOS Monterey. However, as soon as we upgraded to macOS Ventura the computers no longer launch tailscaled on boot or even after logging in.

When we inspect the services that are running we see the following:

user@ventura-host ~ % sudo launchctl list
Password:
PID	Status	Label
...
-	-9	com.tailscale.tailscaled

And when we try to get some info about the service there is the following which seems to indicate some kind of code signing issue?

user@ventura-host ~ % launchctl print system/com.tailscale.tailscaled
system/com.tailscale.tailscaled = {
	active count = 0
	path = /Library/LaunchDaemons/com.tailscale.tailscaled.plist
	type = LaunchDaemon
	state = spawn scheduled

	program = /usr/local/bin/tailscaled
	BTM uuid = 3AC9CA61-D225-4A79-9AC1-E4CAC19A1244
	arguments = {
		/usr/local/bin/tailscaled
	}

	default environment = {
		PATH => /usr/bin:/bin:/usr/sbin:/sbin
	}

	environment = {
		XPC_SERVICE_NAME => com.tailscale.tailscaled
	}

	domain = system
	minimum runtime = 10
	exit timeout = 5
	runs = 15
	last exit reason = OS_REASON_CODESIGNING

	spawn type = daemon (3)
	jetsam priority = 40
	jetsam memory limit (active) = (unlimited)
	jetsam memory limit (inactive) = (unlimited)
	jetsamproperties category = daemon
	jetsam thread limit = 32
	cpumon = default
	probabilistic guard malloc policy = {
		activation rate = 1/1000
		sample rate = 1/0
	}

	properties = keepalive | runatload | inferred program | needs LWCR update | managed LWCR | has LWCR
}

We install tailscale via brew install tailscale. I have just checked the code signature and this is what we see

user@ventura-host ~ % codesign -vvv --deep --verify -d /usr/local/bin/tailscaled
Executable=/usr/local/Cellar/tailscale/1.36.2/bin/tailscaled
Identifier=tailscaled-6013c595efbe000a0fe8d9c677fa7b58fd23092f
Format=Mach-O thin (x86_64)
CodeDirectory v=20400 size=129324 flags=0x2(adhoc) hashes=4035+2 location=system
Hash type=sha256 size=32
CandidateCDHash sha256=bd9055200fd49821d924cee7075d1b63ee28d24d
CandidateCDHashFull sha256=bd9055200fd49821d924cee7075d1b63ee28d24d2bcb925fd303cd378430794e
Hash choices=sha256
CMSDigest=bd9055200fd49821d924cee7075d1b63ee28d24d2bcb925fd303cd378430794e
CMSDigestType=2
Launch Constraints:
	None
CDHash=bd9055200fd49821d924cee7075d1b63ee28d24d
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

On a machine on macOS Monterey we see the following code signature

user@monterey-host ~ % codesign -vvv --deep --verify -d /usr/local/bin/tailscaled
Executable=/usr/local/Cellar/tailscale/1.36.2/bin/tailscaled
Identifier=tailscaled-6013c595efbe000a0fe8d9c677fa7b58fd23092f
Format=Mach-O thin (x86_64)
CodeDirectory v=20400 size=129324 flags=0x2(adhoc) hashes=4035+2 location=system
Hash type=sha256 size=32
CandidateCDHash sha256=bd9055200fd49821d924cee7075d1b63ee28d24d
CandidateCDHashFull sha256=bd9055200fd49821d924cee7075d1b63ee28d24d2bcb925fd303cd378430794e
Hash choices=sha256
CMSDigest=bd9055200fd49821d924cee7075d1b63ee28d24d2bcb925fd303cd378430794e
CMSDigestType=2
CDHash=bd9055200fd49821d924cee7075d1b63ee28d24d
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

I know there are existing tickets (example: https://github.com/tailscale/tailscale/issues/6706) for similar issues but unclear if I should do a “me too” or create a new issue. So went with the new issue!

Steps to reproduce

  • Verify that tailscaled launches at boot on a macOS Monterey 12.6.2 machine
  • Upgrade from macOS Monterey 12.6.2 to macOS Ventura 13.2.1
  • Verify that tailscaled is not launched at boot after upgrade

Are there any recent changes that introduced the issue?

Upgrade to macOS Monterey 13.2.1

OS

macOS

OS version

macOS Ventura 13.2.1

Tailscale version

1.36.2

Other software

No response

Bug report

BUG-be3ac20a0bac40ec5d6da0b036244ff99c1670f4a31ee1680851bfb4edfdc858-20230304205352Z-440f9a893de4fd08

Update 2023-03-06

We just upgraded an Apple Silicon machine and that one does successfully start tailscaled on boot. Below are the successful logs

user@ventura-host-m1 ~ % sudo log show --last boot --debug --info --predicate "eventMessage contains 'tailscale'"
Password:
Filtering the log data using "composedMessage CONTAINS "tailscale""
Timestamp                       Thread     Type        Activity             PID    TTL
2023-03-06 17:36:05.500036-0500 0xd64      Default     0xc49                319    0    backgroundtaskmanagementd: [com.apple.backgroundtaskmanagement:main] registerLaunchItem: found existing item: uuid=14FBA64C-B96C-4D85-95F8-A2E93C6C9625, name=tailscaled, type=legacy daemon, disposition=[enabled, allowed, visible, notified], identifier=com.tailscale.tailscaled, url=file:///Library/LaunchDaemons/com.tailscale.tailscaled.plist
2023-03-06 17:36:05.522402-0500 0xd09      Default     0x0                  1      0    launchd: [system:] Bootstrap by smd[97] for /Library/LaunchDaemons/com.tailscale.tailscaled.plist succeeded (0: )
2023-03-06 17:36:05.700699-0500 0x1114     Default     0x0                  502    0    xpcproxy: Launch constraint set on (null) /Library/LaunchDaemons/com.tailscale.tailscaled.plist
2023-03-06 17:36:05.720184-0500 0x1114     Default     0x0                  0      0    kernel: (AppleMobileFileIntegrity) AMFI: '/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled' has no CMS blob?
2023-03-06 17:36:05.720190-0500 0x1114     Default     0x0                  0      0    kernel: (AppleMobileFileIntegrity) AMFI: '/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled': Unrecoverable CT signature issue, bailing out.
2023-03-06 17:36:05.722987-0500 0xd09      Default     0x0                  1      0    launchd: [system/com.tailscale.tailscaled [502]:] Successfully spawned tailscaled[502] because speculative
2023-03-06 17:36:05.745577-0500 0x9b5      Info        0x2caf               165    0    tccd: [com.apple.TCC:access] AttributionChain: accessing={TCCDProcess: identifier=a.out, pid=502, auid=0, euid=0, binary_path=/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled}, requesting={TCCDProcess: identifier=com.apple.syspolicyd, pid=163, auid=0, euid=0, binary_path=/usr/libexec/syspolicyd},
2023-03-06 17:36:05.745682-0500 0x9b5      Info        0x2caf               165    0    tccd: [com.apple.TCC:access] AttributionChain: accessing={TCCDProcess: identifier=a.out, pid=502, auid=0, euid=0, binary_path=/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled}, requesting={TCCDProcess: identifier=com.apple.syspolicyd, pid=163, auid=0, euid=0, binary_path=/usr/libexec/syspolicyd},
2023-03-06 17:36:05.745693-0500 0x9b5      Default     0x2caf               165    0    tccd: [com.apple.TCC:access] AUTHREQ_ATTRIBUTION: msgID=163.5, attribution={accessing={TCCDProcess: identifier=a.out, pid=502, auid=0, euid=0, binary_path=/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled}, requesting={TCCDProcess: identifier=com.apple.syspolicyd, pid=163, auid=0, euid=0, binary_path=/usr/libexec/syspolicyd}, },
2023-03-06 17:36:05.746393-0500 0x9b5      Info        0x2caf               165    0    tccd: [com.apple.TCC:access] BUNDLE_ATTRIBUTION: executable path file:///opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled resolves to attributed bundle: (null)
2023-03-06 17:36:05.746402-0500 0x9b5      Default     0x2caf               165    0    tccd: [com.apple.TCC:access] AUTHREQ_SUBJECT: msgID=163.5, subject=/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled,
2023-03-06 17:36:05.746610-0500 0x9b5      Default     0x2caf               165    0    tccd: [com.apple.TCC:access] -[TCCDAccessIdentity staticCode]: static code for: identifier /opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled, type: 1: 0x117e0df70 at /opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled
2023-03-06 17:36:05.746724-0500 0x9b5      Info        0x2caf               165    0    tccd: [com.apple.TCC:access] -[TCCDAccessIdentity matchesCodeRequirement:]: SecStaticCodeCheckValidity() static code (0x117e0df70) from /opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled : anchor apple; status: -67050
2023-03-06 17:36:05.746726-0500 0x9b5      Info        0x2caf               165    0    tccd: [com.apple.TCC:access] For /opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled: matches platform requirements: No
2023-03-06 17:36:05.746744-0500 0x9b5      Info        0x2caf               165    0    tccd: [com.apple.TCC:access] Handling access request to kTCCServiceDeveloperTool, from Sub:{/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled}Resp:{TCCDProcess: identifier=a.out, pid=502, auid=0, euid=0, binary_path=/opt/homebrew/Cellar/tailscale/1.36.2/bin/tailscaled}, ReqResult(Auth Right: Unknown (None), DB Action:None, UpdateVerifierData)

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 17 (1 by maintainers)

Most upvoted comments

@amonshiz

I guess I’ll try to get the system to “forget” about the existing tailscaled that was installed by homebrew and then reinstall and see if that makes a difference.

Forgive me if this is a bit pedantic but, do make sure you get all traces of a previous install off the machine(s). NB the system extension can only be removed by a reboot.

+2
quartermarsh on Mar 4, 2023

@quartermarsh building via go build|install is definitely something we will consider, thank you for pointing that out!

At this point it is likely an issue with our configuration or usage. I’m going to try and reproduce in a VM but not sure how well that will go.

Appreciate the help and input! If there is a resolution I’ll post here.

+1
amonshiz on Mar 5, 2023

@amonshiz Any thoughts of bailing on homebrew and just compiling from source following the tailscaled documentation? It sure looks like launchd is unhappy with where homebrew is putting the launch daemon and Security Policy is throwing a block so tailscale won’t launch at boot. You’d think you’d get a prompt to allow tailscale in Security and Privacy if that is the case. Of course I admit that I might be misreading this altogether.

If you do decide to compile from source, go will install in your home folder by default. From there its a matter of tweaking your $PATH and your shell script (and maybe allowing full disk access for terminal, although I can’t be sure about that off the top of my head). Of course I may not be anticipating every complication but it looks like it might be worth a shot at this point.

+1
quartermarsh on Mar 5, 2023

I just tried completely removing Tailscale from the system again and then ensuring homebrew is actually building the formulae instead of installing a “bottle”:

user@ventura-host ~ % brew install -s tailscale
Warning: Treating tailscale as a formula. For the cask, use homebrew/cask/tailscale
==> Fetching dependencies for tailscale: go
==> Fetching go
==> Downloading https://ghcr.io/v2/homebrew/core/go/manifests/1.20.1
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/go/blobs/sha256:b763597544bbe1df40a63ef2bbe10c04855d674e65322efa0c97fc4938ff5bd8
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:b763597544bbe1df40a63ef2bbe10c04855d674e65322efa0c97fc49
######################################################################## 100.0%
==> Fetching tailscale
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/3e1ff979e76a8e77c607fa9d59ba7ac911f8d666/Formula/tailscale.rb
######################################################################## 100.0%
==> Cloning https://github.com/tailscale/tailscale.git
Cloning into '/Users/user/Library/Caches/Homebrew/tailscale--git'...
==> Checking out tag v1.36.2
HEAD is now at 0438c67e VERSION.txt: this is v1.36.2
==> Installing dependencies for tailscale: go
==> Installing tailscale dependency: go
==> Pouring go--1.20.1.ventura.bottle.tar.gz
🍺  /usr/local/Cellar/go/1.20.1: 11,968 files, 233.8MB
==> Installing tailscale
==> go build -ldflags=-s -w -X tailscale.com/version.Long=1.36.2-t0438c67e2 -X tailscale.com/version.Short=1.36.2 -X tailscale.com/version.Gi
==> go build -ldflags=-s -w -X tailscale.com/version.Long=1.36.2-t0438c67e2 -X tailscale.com/version.Short=1.36.2 -X tailscale.com/version.Gi
==> Caveats
To restart tailscale after an upgrade:
  brew services restart tailscale
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/tailscale/bin/tailscaled
==> Summary
🍺  /usr/local/Cellar/tailscale/1.36.2: 9 files, 26.3MB, built in 26 seconds
==> Running `brew cleanup tailscale`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Caveats
==> tailscale
To restart tailscale after an upgrade:
  brew services restart tailscale
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/tailscale/bin/tailscaled

I then enabled the service by putting the recommended plist file in /Library/LaunchDaemons, starting the service, logging into Tailscale (tailscale up ...), and then rebooting the computer. Still Tailscale did not start at boot. More of the same error logs:

2023-03-04 20:42:52.669337-0800 0x6d1      Default     0x0                  0      0    kernel: (AppleMobileFileIntegrity) AMFI: Launch Constraint Violation (enforcing), error info: c[5]p[1]m[5]e[5], (cdhash) launching proc[vc: 0 pid: 298]: /usr/local/Cellar/tailscale/1.36.2/bin/tailscaled, launch type 0, failure proc [vc: 0 pid: 298]: /usr/local/Cellar/tailscale/1.36.2/bin/tailscaled
2023-03-04 20:42:52.669431-0800 0x6db      Default     0x0                  0      0    kernel: (AppleSystemPolicy) ASP: log-executable mig callout failed 0x10000004 (path: /usr/local/Cellar/tailscale/1.36.2/bin/tailscaled responsiblePath: /usr/local/Cellar/tailscale/1.36.2/bin/tailscaled isExecution: 1)
2023-03-04 20:42:52.669463-0800 0x6db      Default     0x0                  0      0    kernel: (AppleSystemPolicy) ASP: Security policy would not allow process: 298, /usr/local/Cellar/tailscale/1.36.2/bin/tailscaled

Update: and now launchd does not launch tailscale at boot again, even after resigning. 😦

+1
amonshiz on Mar 5, 2023

@amonshiz Compiling with go is usually ‘go install’ for the git pull and then ‘go build’. It looks like that is what the homebrew formula is doing.

+1
quartermarsh on Mar 5, 2023

@amonshiz Well, I’m glad it works. As for the differences in the installs, I likewise have no idea. I would lean towards it having something to do with how Ventura installs binaries. Maybe the SDK changed? I guess it ain’t broke for now, which is good.

@quartermarsh I would not quantify forcing code signing as it working. 😕 I am trying to upgrade our CI fleet and having to go in to each host to sign and restart the service is not sustainable. I am not sure if this would break on the next Tailscale update. 😦

On the flip side, we have not heard of any of our employee machines having an issue when upgrading to Ventura in the past months.

+1
amonshiz on Mar 5, 2023

Do the Zip files in https://pkgs.tailscale.com/stable/#macos have the same codesigning issue? We’ve been able to run the SystemExtension app on Ventura here without trouble.

That Zip file is what brew sources for their installation.

+1
DentonGentry on Mar 5, 2023

@amonshiz

I do not believe there are any system extensions on the system

Sorry, I meant the tailscale extension.

+1
quartermarsh on Mar 4, 2023
Read more comments on GitHub
← tailscale: `tailscale ssh` doesn't work with the macOS sandbox app
tailscale: tailscaled doesn't start on Google Colaboratory →