kubernetes: Invalid Error: .container.volumes: hostPath type check failed: "not a directory"

I had the same issue using minikube and managed to solve it by starting minikube with --mount and --mount-string flags

minikube start --mount --mount-string="/host/path:/minikubeVM/path"

I got the exact same error as you showed.

In my case, the issue was the folder defined in volume hostPath was not created. Once the folder was created in the worker node server, the issue was addressed.

I facing the same issue and it’s take 3 days. finally i solved by adding :

type: DirectoryOrCreate
path: /run/desktop/mnt/host/c/users/public/your_folder

Yeah you have to remove the type field and it’s works 😃 Le sam. 9 mai 2020 à 02:23, Mithil Patel notifications@github.com a écrit : … Was there a solution to this issue? I’m facing this issue as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#83125 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFM3IPLCPGW5O2SRDXAZ33LRQSPA7ANCNFSM4I2PWVLA .

No it doesn’t work correctly. If we remove the type file it doesn’t give the error but it also doesn’t mount the directory which exists on the host. When i go into the directory on the pod, it is empty.

I am able to reproduce this issue using the csi-cinder-plugin manifest here: https://github.com/kubernetes/cloud-provider-openstack/blob/master/manifests/cinder-csi-plugin/cinder-csi-nodeplugin.yaml#L95-L98

volumes:
        - name: pods-cloud-data
          hostPath:
            path: /var/lib/cloud/data
            type: Directory

yields

Warning  FailedMount  8s (x9 over 2m16s)  kubelet, oscsi-kworker-green-0  MountVolume.SetUp failed for volume "pods-cloud-data" : hostPath type check failed: /var/lib/cloud/data is not a directory

but I can verify the directory exists on the host machine

[centos@oscsi-kworker-green-0 ~]$ ls -la /var/lib/cloud/data
total 28
drwxr-xr-x. 2 root root 163 Jan  8 01:32 .
drwxr-xr-x. 8 root root 105 Jan  8 01:32 ..
-rw-r--r--. 1 root root  37 Jan  8 01:32 instance-id
-rw-r--r--. 1 root root  63 Jan  8 01:32 previous-datasource
-rw-r--r--. 1 root root  42 Dec  3 21:52 previous-hostname
-rw-r--r--. 1 root root  37 Jan  8 01:32 previous-instance-id
-rw-r--r--. 1 root root  88 Jan  8 01:32 result.json
-rw-r--r--. 1 root root  86 Jan  8 01:32 set-hostname
-rw-r--r--. 1 root root 600 Jan  8 01:32 status.json

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

I am running Kubernetes on CentOS7, running on OpenStack. Removing type: Directory (defaulting to DirectoryOrCreate) is a workaround that works for me, but this doesn’t appear to always be working as intended.

I believe that directory is created by cloud-init, if that’s relevant.

A solution that worked for me…

macOS Ventura 13.3.1
Apple M1 (arm64)
Docker version 23.0.5, build bc4487a
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

Following the advice seen elsewhere, attempted to use a /data/ directory on the HOST MACHINE. When specifying type: Directory for the hostPath pv this would fail, as seen on this thread. Upon removing the type: Directory restriction pods would successfully start, but then as noted by others changes to the mount path inside the pod were NOT reflected in the host (macOS) environment.

As required, we can modify the “shared files” in Docker Desktop (Settings -> Resources -> Shared Files) and, following the advice seen elsewhere attempt to share our HOST MACHINE /data/ directory with Docker. Then minikube delete ; minikube start --mount-string="/data:/data" --mount – however, going into a terminal on the container and performing cd /data; touch x; ls -al shows a file named x is created inside /data/ directory within the minikube container, but /data/ on the HOST MACHINE remains empty. This, of course, is before even involving a hostPath pv via Minikube!

Trying again, minikube delete ; minikube start --mount-string="/data:/test" – then back into terminal and ls -al / to see that there are both /data/ and /test/ at the volume root. Take note, as this highlights the problem. Now cd /test/ ; touch x ; ls -al and see x file is created in /test/ within the minikube container, in the HOST MACHINE /data/ also see that x is created.

The problem is the bind mount from Docker is not applied to the existing /data/ in the container image. It doesn’t matter what directory names you use on the HOST MACHINE this doesn’t seem to work as described.

The reason type: Directory fails is because /data/ does not bind to your mount as specified on the CLI, and, the expected directories obviously do not exist. The reason type: DirectoryOrCreate (or removal of type:) appears to work is because the /data/ directory exists, and, the path specified within the hostPath spec is successfully created by minikubes hostPath implementation. The obvious problem is that /data/ isn’t bound to the HOST MACHINE by docker as is expected (because /data/ already exists within the minikube container image!), and therefore is not retained outside of the minikube container.

Other notes:

I do not use virtualbox for VM on macOS (unnecessary dependency.)

This problem was present whether I used gRPC FUSE or VirtioFS in Docker engine.

---

MY ADVICE TO OTHER MINIKUBE USERS

1. do not use /data/ as bind path, use something unique that will avoid collision with minikube project, in my case i have gone with /d34db33f/ – lel.
2. when starting a minikube instance, i can still use my host /data/ dir Fx.: minikube start --mount-string="/data:/d34db33f"
3. in my /data/ on the HOST MACHINE i create my-subfolder/
4. in my pv i specify path: /d34db33f/my-subfolder and type: Directory
5. in my container spec i specify a mount path of /data/my-subfolder – i achieve parity between HOST MACHINE and POD environment.
6. i rollout a deployment and everything works, files appear within /data/ on HOST MACHINE as they are created within minikube container.

Alternatively, don’t bind to /data/ and instead bind to specific sub-paths under /data/ for use in your pv specs – yes, it’s more work, and it’s not something I want to be dealing with either… but in theory a mount string such as “/data/foo:/data/foo” would work as expected and is probably why minikube container image has a /data/ in the first place. The flaw with this approach is the obvious restriction that once the minikube container is created in Docker engine new/different mount paths cannot be specified, making it a frustrating approach.

---

Hope this helps others, i doubt this is OS/Platform/Docker/Minikube specific but have provided all versions I have in use anyway, I suspect this has been plaguing people over all versions for several years because the root cause was not well understood, and if anyone understood it they very, VERY poorly explained it. OP was 2019, it is now 2023.

NOTE: I have to point out that the behavior described by OP is “working as intended, by design” but the reason this is the case was not explained.

👍

I’m trying to figure out why this issue was closed. It appears to be a real bug.

This issue should be reopened unless it is being tracked elsewhere.

@ruizmarlon I actually think I was wrong, I forgot I commented here. I was using microk8s and I needed to set the kubelet folder path to the one in snap since microk8s is a snap. Maybe it’s a real fix for you but just want to give you a heads up.

^ @stephen304 adding those directories worked like magic. Was using k0s & ceph-rook for my experiment. Thanks!!!

I just had this issue and fixed it by creating the 2 folders referenced in the error. Took me a while to find the right logs:

/var/lib/kubelet/pods/
/var/lib/kubelet/plugins_registry/

Yeah you have to remove the type field and it’s works 😃

Le sam. 9 mai 2020 à 02:23, Mithil Patel notifications@github.com a écrit :

Was there a solution to this issue? I’m facing this issue as well.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes/issues/83125#issuecomment-626073653, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFM3IPLCPGW5O2SRDXAZ33LRQSPA7ANCNFSM4I2PWVLA .