longhorn: Failed to get arg root-dir: Cannot get kubelet root dir, no related proc for root-fir detection, error out

Im seeing following error on openshift 3.11 (i.e. k8s 1.11)

oc logs -f longhorn-driver-deployer-59bd589d86-84k24

time="2019-03-23T14:28:33Z" level=debug msg="Driver csi will be used after automatic detection"
--
  | time="2019-03-23T14:28:33Z" level=debug msg="Deploying CSI driver"
  | time="2019-03-23T14:28:33Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:34Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:36Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:37Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:38Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:40Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:41Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:42Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:43Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:45Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:46Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
  | time="2019-03-23T14:28:48Z" level=error msg="Failed to get arg root-dir: Cannot get kubelet root dir, no related proc for root-fir detection, error out"
  | time="2019-03-23T14:28:48Z" level=fatal msg="Error deploying driver: Failed to get arg root-dir: Cannot get kubelet root dir, no related proc for root-fir detection, error out"

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 37 (13 by maintainers)

Most upvoted comments

I found out that kubelet seems unable to communicate with CSI plugin at all, except for creation or attaching. So I took a second look at the document you provided and noticed this:

        - name: socket-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi-cinderplugin
            type: DirectoryOrCreate
        - name: mountpoint-dir
          hostPath:
            path: /var/lib/origin/openshift.local.volumes/pods/
            type: Directory

So it seems OpenShift didn’t change kubelet root-dir in the end, but it changed the mountpoint-dir to /var/lib/origin/openshift.local.volumes/pods/, which is really unusual. That’s why I always wondering why the root-dir of kubelet is /var/lib/origin/openshift.local.volumes/. In the end, it’s not. It’s only the path for mount. Kubelet root dir is still /var/lib/kubelet/ for OpenShift. So the plugins should be installed in /var/lib/kubelet/plugins/io.rancher.longhorn/csi.sock but mountpoint-dir should be /var/lib/origin/openshift.local.volumes/pods/.

Current Longhorn doesn’t provide an option to update the mountpoint-dir, which is <root-dir>/pods. We can add that option later to support OpenShift.

There is probably a workaround:

Redeploy Longhorn, and set root-dir to /var/lib/kubelet. Still using CSI.
On every node, run mount --bind /var/lib/origin/openshift.local.volumes/pods /var/lib/kubelet/pods

This may work.

Or you can try using Flexvolume driver. Just set the driver to flexvolume and don’t set other options to see if it works.

What’s the kubelet command line for OpenShift? I’d like to see that for reference.

yasker on Apr 2, 2019

I also have this issue with longhorn0.4.1 on DigitalOcean cluster 1.13.5.do. Any idea how to fix this?

time="2019-03-28T12:00:47Z" level=debug msg="User specified the driver csi"
time="2019-03-28T12:00:47Z" level=debug msg="Deploying CSI driver"
time="2019-03-28T12:00:47Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
time="2019-03-28T12:00:48Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
time="2019-03-28T12:00:49Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
time="2019-03-28T12:00:50Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Running"
time="2019-03-28T12:00:51Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
time="2019-03-28T12:00:52Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
time="2019-03-28T12:00:53Z" level=debug msg="kubelet arg detection pod discover-kubelet-root-dir in phase: Pending"
time="2019-03-28T12:00:54Z" level=error msg="Failed to get arg root-dir: Cannot get kubelet root dir, no related proc for root-fir detection, error out"
time="2019-03-28T12:00:54Z" level=fatal msg="Error deploying driver: Failed to get arg root-dir: Cannot get kubelet root dir, no related proc for root-fir detection, error out"

botzill on Mar 28, 2019