tailscale: network-lock unavailable; no state directory
What is the issue?
I am observing two behaviours that might be related.
I am not 100% sure if these are related things yet, but I suspect they are.
tailscaledconstantly logging “network-lock unavailable; no state directory”- AWS Systems Manager Parameter Store being updated many times
Log Snippet
Aug 03 16:58:56 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:03:46 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:18:34 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:24:12 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:39:27 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:40:56 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:44:23 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:45:23 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:46:01 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:46:02 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:46:17 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:46:17 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:46:18 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 17:54:25 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 18:15:39 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 18:19:39 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 18:29:24 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 18:29:24 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 18:30:25 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
Aug 03 18:30:44 ip-10-0-1-13.eu-west-1.compute.internal tailscaled[2779]: network-lock unavailable; no state directory
AWS Systems Manager Parameter Store Screenshot
Steps to reproduce
Launch an EC2 instance with “Amazon Linux 2” (latest, from the official parameter store AMI).
Do a yum install tailscale.
Using almost default tailscaled.service: https://github.com/tailscale/tailscale/blob/main/cmd/tailscaled/tailscaled.service
With the following change:
sed -i 's|/var/lib/tailscale/tailscaled.state|arn:aws:ssm:eu-west-1:123:parameter/CFN-TailscaleState|g' \
/usr/lib/systemd/system/tailscaled.service
Resulting tailscaled.service
[Unit]
Description=Tailscale node agent
Documentation=https://tailscale.com/kb/
Wants=network-pre.target
After=network-pre.target NetworkManager.service systemd-resolved.service
[Service]
EnvironmentFile=/etc/default/tailscaled
ExecStartPre=/usr/sbin/tailscaled --cleanup
ExecStart=/usr/sbin/tailscaled --state=arn:aws:ssm:eu-west-1:123:parameter/CFN-TailscaleState --socket=/run/tailscale/tailscaled.sock --port=${PORT} $FLAGS
ExecStopPost=/usr/sbin/tailscaled --cleanup
Restart=on-failure
RuntimeDirectory=tailscale
RuntimeDirectoryMode=0755
StateDirectory=tailscale
StateDirectoryMode=0700
CacheDirectory=tailscale
CacheDirectoryMode=0750
Type=notify
[Install]
WantedBy=multi-user.target
Are there any recent changes that introduced the issue?
Not sure.
OS
Linux
OS version
Amazon Linux 2
Tailscale version
1.46.1
Other software
No response
Bug report
BUG-79aa465587e8bd7604513f5b5101001f7b1a637933644fad6216a2d7efbe2840-20230803190443Z-c006b4f6f1c60cb6
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 20 (7 by maintainers)
Commits related to this issue
- ipn/store/awsstore: skip, flag useless writes DO NOT SUBMIT Updates #8785 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> — committed to tailscale/tailscale by bradfitz a year ago
- ipn: avoid useless no-op WriteState calls Rather than make each ipn.StateStore implementation guard against useless writes (a write of the same value that's already in the store), do writes via a new... — committed to tailscale/tailscale by bradfitz a year ago
- ipn: avoid useless no-op WriteState calls Rather than make each ipn.StateStore implementation guard against useless writes (a write of the same value that's already in the store), do writes via a new... — committed to tailscale/tailscale by bradfitz a year ago
It should work, but test away.
We do a release about monthly.
@moltar, I just merged https://github.com/tailscale/tailscale/pull/8793