k8s-on-raspbian: Failed to find subsystem mount for required subsystem: pids

Expected Behaviour

kubernetes master node starts and kubectl get pods shows Ready status.

Current Behaviour

kubernetes master node starts, and kubectl get pods shows NotReady status

kubectl describe nodes shows this error in the event log: Failed to update Node Allocatable Limits [“kubepods”]: failed to set supported cgroup subsystems for cgroup [kubepods]: Failed to find subsystem mount for required subsystem: pids

Possible Solution

not sure - disable whatever requires this cgroup? is it something new in 1.14? or enable that cgroup in rasberian lite somewhere? (I’m not cgroup expert, so I don’t know how to even start)

Steps to Reproduce (for bugs)

(follow the guide in this repo, I get these results at the “Check everything worked:” step of the guide)

Context

Can’t schedule pods / nodes not ready.

Your Environment

  • Docker version docker version (e.g. Docker 17.0.05 ): Docker version 18.09.0, build 4d60db4

  • What version of Kubernetes are you using? kubectl version: Client Version: version.Info{Major:“1”, Minor:“14”, GitVersion:“v1.14.0”, GitCommit:“641856db18352033a0d96dbc99153fa3b27298e5”, GitTreeState:“clean”, BuildDate:“2019-03-25T15:53:57Z”, GoVersion:“go1.12.1”, Compiler:“gc”, Platform:“linux/arm”} Server Version: version.Info{Major:“1”, Minor:“14”, GitVersion:“v1.14.0”, GitCommit:“641856db18352033a0d96dbc99153fa3b27298e5”, GitTreeState:“clean”, BuildDate:“2019-03-25T15:45:25Z”, GoVersion:“go1.12.1”, Compiler:“gc”, Platform:“linux/arm”}

  • Operating System and version (e.g. Linux, Windows, MacOS): 2018-11-13-raspbian-stretch-lite.img 4.14.79-v7+

  • What ARM or Raspberry Pi board are you using? Rasberry Pi 3 B+

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 9
  • Comments: 22 (4 by maintainers)

Most upvoted comments

PIDs cgroup will be available on the next rpi release: https://github.com/raspberrypi/linux/pull/2968#issuecomment-492552354.

I’ve managed to upgrade my raspbian to buster, but it wasn’t error free, the following issues where hit:

iptables in nf_tables mode - kube-proxy only works in legacy mode https://github.com/kubernetes/kubernetes/issues/71305#issuecomment-479558920

swap would be enabled after each boot: sudo systemctl disable dphys-swapfile followed by a reboot, fixed this.

I noticed a substantial amount of errors being reported from docker reporting that cgroupsfs/net_prio being missing (even though it existed and mounted) - upgrading docker-ce to 18.09.0 resolved these - in addition - I had to install docker from stretch repo as busters is known to be broken: https://github.com/docker/for-linux/issues/709

@Mike-Dunton I was able to rpi-update to 4.19.46-v7+ today, and confirm the PIDS fix is in place. kubeadm 1.14.3 installs and inits fine.

@davidcollom Thank you for those hints. Didn’t notice that iptables wasn’t working correctly but fixed it now with the command from your link.

I had the same issue. Did a dist-upgrade from stretch to buster and now kubernetes version v1.15.0 works. Buster has kernel version 4.19.50-v7+ and includes the PIDS Cgroup. However there is a warning when running kubelet describe node master that says kubelet is posting ready status. WARNING: CPU hardcapping unsupported. I’m not sure if this bothers kubernetes but since it’s running fine i guess not.

Note: After upgrading in place i had to disable swap again.

Thanks hgontijo to push for it 😃

Recompiling Raspbian kernel with CONFIG_CGROUP_PIDS enabled fixed this issue for me (I was running k8s v1.14.1)