kops: Kubelet service breaks after reboot of GCE nodes (permission denied)

1. What kops version are you running? The command kops version, will display this information.

Version 1.8.1

1. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

1.8.7 and 1.8.8

1. What cloud provider are you using?

GCE

1. What commands did you run? What is the simplest way to reproduce this issue?

After the initial node boot, we’re able to run /home/kubernetes/bin/kubelet -h and the kubelet service starts up fine, however if the node is rebooted (GCE maintenance or manually running sudo shutdown -r now) the kubelet service fails to start and we’re unable to run it manually (even with sudo), getting permission denied.

systemd[20259]: kubelet.service: Failed at step EXEC spawning /home/kubernetes/bin/kubelet: Permission denied

and

 $ sudo /home/kubernetes/bin/kubelet 
sudo: unable to execute /home/kubernetes/bin/kubelet: Permission denied

1. What happened after the commands executed?

Get permission denied.

1. What did you expect to happen?

The kubelet service to start after the node is rebooted

1. Please provide your cluster manifest. Execute kops get --name my.example.com -oyaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

n/a

1. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

n/a

1. Anything else do we need to know?

We also have k8s clusters provisioned via kops in AWS and confirmed we are able to reboot nodes and afterwards they return to Ready state without issue.

Before rebooting the instance has the following mount listed:

$ mount | grep kubernetes/bin
/dev/sda1 on /home/kubernetes/bin type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)

and it does not show up after the reboot. I’m trying to dig in to how Kops / Container Optimized OS manage these but assuming it’s related to something along those lines. If the issue lies with Container Optimized OS, apologies for opening this ticket, but any help you can provide to work around or resolve it would still be much appreicated.

I should mention the file is still “there”, and the permissions show anyone should be able to execute it:

$ ls -alh /home/kubernetes/bin/kubelet 
-rwxr-xr-x 1 root root 132M Apr 12 00:38 /home/kubernetes/bin/kubelet