node-driver-registrar: "Failed to create registration probe file" with readOnlyRootFilesystem

When using readOnlyRootFilesystem: true the node-driver-registrar tries to recursively create the directory the probe file is in, leading to an error:

E0804 18:06:27.375117       1 main.go:107] "Failed to create registration probe file" err="mkdir /var/lib/kubelet: read-only file system" registrationProbePath="/var/lib/kubelet/plugins/ebs.csi.aws.com/registration"

In this scenario /var/lib/kubelet/plugins/ebs.csi.aws.com/ is a hostPath mounted into the container (and registration works just fine), but node-driver-registrar tries to create /var/lib/kubelet/ despite the fact it already exists and fails, never creating the probe file.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 3
Comments: 18 (9 by maintainers)

Most upvoted comments

@yogeek I tried to also make node-driver-registrar compatible with readOnlyRootFilesystem: true in https://github.com/kubernetes-csi/node-driver-registrar/pull/214 but it created other problems so I had to roll it back.

The latest versions of node-driver-registrar should have the issue in https://github.com/kubernetes-csi/node-driver-registrar/issues/213#issue-1329190786 which only happens if the Deployment has readOnlyRootFilesystem: true but it shouldn’t block the deployment of a CSI Driver if you don’t need this feature.

mauriciopoppe on Apr 14, 2023

Hi @mauriciopoppe I just installed aws-ebs-csi-driver on our k8s 1.22 to prepare for the CSI migration needed to upgrade to 1.23 and the node-driver-registrar logs still have this error

As the #214 fix was reverted, what will be the next step to fix this please ? As I am not very familiar with this component, is this issue is preventing the driver to work properly ?

(I used the helm chart aws-ebs-csi-driver-2.17.2 which installed the 1.17.0 app version including the public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar:v2.7.0-eks-1-26-latest image)

UPDATE : it does not prevent the driver to work, I just tested on a 1.22 cluster and the volumes are created successfully

yogeek on Mar 29, 2023