k3s: ISCSI mount fails, but can be fixed manually

Awesome project, though I am having some trouble with iscsi:

Describe the bug When a pod is created with a iscsi volume, the pod will fail to mount the block. Running the command from the event works and allows pod to succeed.

To Reproduce

run iscsiadm -m discover -t sendtargets -p $IP on kube worker node. – assume pre-formatted block device to ext4
create with a iscsi volume, as per the example here. Obviously replace connection info with correct settings. – I am using a synology iscsi target
error shows:

  Warning  FailedMount  2m5s   kubelet, lab02.libretinker.com  MountVolume.SetUp failed for volume "iscsipd-rw" : mount failed: exit status 1
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/rancher/k3s/agent/kubelet/pods/42351a5d-3c02-11e9-87ea-b827eb93df71/volumes/kubernetes.io~iscsi/iscsipd-rw --scope -- mount -o bind /var/lib/rancher/k3s/agent/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.1.2:3260-iqn.2000-01.com.synology:hds.Target-21.336328a62a-lun-1 /var/lib/rancher/k3s/agent/kubelet/pods/42351a5d-3c02-11e9-87ea-b827eb93df71/volumes/kubernetes.io~iscsi/iscsipd-rw
Output: Running scope as unit: run-r430a1e406f8f488da063a8c2ee197b9a.scope
mount: bad address '/var/lib/rancher/k3s/agent/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.1.2'

running the following on the assigned worker node will allow the pod to successfully start after it tries again: (note that is is mkdir followed by the failed command in the error)

$ mkdir /var/lib/rancher/k3s/agent/kubelet/pods/42351a5d-3c02-11e9-87ea-b827eb93df71/volumes/kubernetes.io~iscsi/iscsipd-rw
$ systemd-run --description="Kubernetes transient mount for /var/lib/rancher/k3s/agent/kubelet/pods/42351a5d-3c02-11e9-87ea-b827eb93df71/volumes/kubernetes.io~iscsi/iscsipd-rw" --scope -- mount -o bind /var/lib/rancher/k3s/agent/kubelet/plugins/kubernetes.io/iscsi/iface-default/192.168.1.2:3260-iqn.2000-01.com.synology:hds.Target-21.336328a62a-lun-1 /var/lib/rancher/k3s/agent/kubelet/pods/42351a5d-3c02-11e9-87ea-b827eb93df71/volumes/kubernetes.io~iscsi/iscsipd-rw

Expected behavior mount should bind to pod directory and the pod should start on its own

Screenshots n/a

Additional context

I assume the same issue would exist with PVs but I did not test that - the kubernetes example just uses a pod with a volume.
worker nodes are fedora, master node is raspbian.

Thanks!

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 19 (4 by maintainers)

Most upvoted comments

Sorry @lentzi90 the latest RC doesn’t have the fix, but will be available in the next RC!

deniseschannon on May 3, 2019

@lentzi90 v0.5.0-rc4 is now available to test!

deniseschannon on May 4, 2019

I also ran into this issue on Arch with k3s + openebs, symlinking the k3s mount binary to the system one did the trick and now my pvc is mounted as expected. ln -sf /usr/sbin/mount /var/lib/rancher/k3s/data/{hash}/bin/mount

pysen on Apr 5, 2019

After further testing, this seems to effect NFS mounts as well (all remote mounts?).

  Warning  FailedMount       6s (x6 over 22s)   kubelet, lab01     MountVolume.SetUp failed for volume "pvc-2e15f26e-3d80-11e9-8ce6-b827eb93df71" : mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs 192.168.1.2:/volume1/AppStoreNG/default-slzc-pvc-2e15f26e-3d80-11e9-8ce6-b827eb93df71 /var/lib/rancher/k3s/agent/kubelet/pods/2e18a4f9-3d80-11e9-8ce6-b827eb93df71/volumes/kubernetes.io~nfs/pvc-2e15f26e-3d80-11e9-8ce6-b827eb93df71
Output: mount: mounting 192.168.1.2:/volume1/AppStoreNG/default-slzc-pvc-2e15f26e-3d80-11e9-8ce6-b827eb93df71 on /var/lib/rancher/k3s/agent/kubelet/pods/2e18a4f9-3d80-11e9-8ce6-b827eb93df71/volumes/kubernetes.io~nfs/pvc-2e15f26e-3d80-11e9-8ce6-b827eb93df71 failed: Connection refused

If I run the mount command from the error event, it fails - but if I run mkdir first, it will succeed - and the pod will start correctly. This is exactly the behavior of the iscsi issue on a systemd host.

Could the issue be as simple as the mounting code needing to run os.Mkdir() before it mounts?

edit: also reproduced on master (currently f90cbed4081e7e1e6972861c196543b2d253bfcc)

iluminae on Mar 3, 2019