k3s: Fresh K3s install fails to start with \"k3s-server\": executable file not found in $PATH

Environmental Info: K3s Version:

k3s version v1.20.2+k3s1 (1d4adb03)
go version go1.15.5

Node(s) CPU architecture, OS, and Version:

Linux msl-k3s3 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.9.2009 (Core)

Cluster Configuration: 1 node

Describe the bug: We deploy a few hundred fresh test instance a day, Centos and puppet based. It is a completely clean automated install. ~10% of the k3s installations fail with the following error:

16:21:57 Error: Systemd start for k3s failed!
16:21:57 journalctl log for k3s:
16:21:57 -- Logs begin at Thu 2021-01-21 16:06:15 CET, end at Thu 2021-01-21 16:21:57 CET. --
16:21:57 Jan 21 16:21:56 msl-k3s3 systemd[1]: Starting Lightweight Kubernetes...
16:21:57 Jan 21 16:21:57 msl-k3s3 k3s[455]: time="2021-01-21T16:21:57+01:00" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: Failed to start Lightweight Kubernetes.
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: Unit k3s.service entered failed state.
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: k3s.service failed.

I can see that not all k3s files have been populated properly. There is plenty of space, all the instances are the same.

Systemd k3s service:

# /etc/systemd/system/k3s.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/k3s.env
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/k3s \
    server \
	'--docker' \
	'--no-deploy=traefik' \
	'--no-deploy=servicelb' \
	'--no-deploy=local-storage' \
	'--no-deploy=metrics-server' \
	'--cluster-cidr=172.16.0.0/14' \
	'--service-cidr=172.20.0.0/16' \

KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

Steps To Reproduce: It is random as far as I can tell. ~10% installations fail.

Expected behavior: K3s always starts ok.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

For anyone else coming across this issue. Check in /etc/fstab that your /var mount does NOT have the noexec option. If it does delete that option and reboot. I had the same issue on RedHat 8. The servers I was using had been hardened to NIST800 standard, so not sure if noexec is a default option for redhat on /var.