k3s: Fresh K3s install fails to start with \"k3s-server\": executable file not found in $PATH
Environmental Info: K3s Version:
k3s version v1.20.2+k3s1 (1d4adb03)
go version go1.15.5
Node(s) CPU architecture, OS, and Version:
Linux msl-k3s3 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.9.2009 (Core)
Cluster Configuration: 1 node
Describe the bug: We deploy a few hundred fresh test instance a day, Centos and puppet based. It is a completely clean automated install. ~10% of the k3s installations fail with the following error:
16:21:57 Error: Systemd start for k3s failed!
16:21:57 journalctl log for k3s:
16:21:57 -- Logs begin at Thu 2021-01-21 16:06:15 CET, end at Thu 2021-01-21 16:21:57 CET. --
16:21:57 Jan 21 16:21:56 msl-k3s3 systemd[1]: Starting Lightweight Kubernetes...
16:21:57 Jan 21 16:21:57 msl-k3s3 k3s[455]: time="2021-01-21T16:21:57+01:00" level=fatal msg="exec: \"k3s-server\": executable file not found in $PATH"
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: Failed to start Lightweight Kubernetes.
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: Unit k3s.service entered failed state.
16:21:57 Jan 21 16:21:57 msl-k3s3 systemd[1]: k3s.service failed.
I can see that not all k3s files have been populated properly. There is plenty of space, all the instances are the same.
Systemd k3s service:
# /etc/systemd/system/k3s.service
[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
After=network-online.target
[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/k3s.env
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/bin/k3s \
server \
'--docker' \
'--no-deploy=traefik' \
'--no-deploy=servicelb' \
'--no-deploy=local-storage' \
'--no-deploy=metrics-server' \
'--cluster-cidr=172.16.0.0/14' \
'--service-cidr=172.20.0.0/16' \
KillMode=process
Delegate=yes
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
Steps To Reproduce: It is random as far as I can tell. ~10% installations fail.
Expected behavior: K3s always starts ok.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (9 by maintainers)
For anyone else coming across this issue. Check in /etc/fstab that your /var mount does NOT have the
noexecoption. If it does delete that option and reboot. I had the same issue on RedHat 8. The servers I was using had been hardened to NIST800 standard, so not sure if noexec is a default option for redhat on /var.