aws-efs-csi-driver: EFS CSI driver can't mount successfully, keeps retrying

/kind bug

What happened? After a period of time where it is working successfully, an EFS CSI stops mounting new EFS volumes, but it keeps trying, resulting in dozens of stunnel instances that appear to be stuck.

I have a single pod in testing that creates 4 mounts on a single EFS volume (using subpaths), and I’ve been tearing that pod down and recreating it.

On the affected node, I see no active nfs/efs mounts:

root@ip-10-104-148-31:/home/admin# mount | grep nfs
root@ip-10-104-148-31:/home/admin#
---
root@ip-10-104-148-31:/home/admin# mount | grep fs-734a4047
root@ip-10-104-148-31:/home/admin#

But I do see a whole heap of attempted mounts for that one volume:

root@ip-10-104-148-31:/home/admin# ps -ef | grep mount
root       806 17844  0 20:53 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root       807   806  0 20:53 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root       809   807  0 20:53 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.2d954c71-b531-4c30-929a-085e636c63c3.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20373
root       876   807  0 20:53 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20373,nodev
root      2821 17844  0 20:54 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--private/mount
root      2822  2821  0 20:54 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root      2825 17844  0 20:54 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount
root      2826  2825  0 20:54 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root      2828  2822  0 20:54 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.2d954c71-b531-4c30-929a-085e636c63c3.volumes.kubernetes.io~csi.au10025--dev--private.mount.20431
root      2829  2826  0 20:54 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.2d954c71-b531-4c30-929a-085e636c63c3.volumes.kubernetes.io~csi.au10025--dev--public.mount.20091
root      3033  2822  0 20:54 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20431,nodev
root      3035  2826  0 20:54 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20091,nodev
root      3769 17844  0 20:55 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root      3770  3769  0 20:55 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root      3781  3770  0 20:55 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.2d954c71-b531-4c30-929a-085e636c63c3.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20094
root      3783  3770  0 20:55 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20094,nodev
root      5663 17844  0 20:56 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
root      5664  5663  0 20:56 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root      5667 17844  0 20:56 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
root      5668  5667  0 20:56 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root      5847  5664  0 20:56 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--private.mount.20326
root      5848  5668  0 20:56 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--public.mount.20248
root      5850  5664  0 20:56 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20326,nodev
root      5852  5668  0 20:56 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20248,nodev
root      6538 17844  0 20:57 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root      6539  6538  0 20:57 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root      6541  6539  0 20:57 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20174
root      6553  6539  0 20:57 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20174,nodev
root      8547 17844  0 20:58 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
root      8548  8547  0 20:58 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root      8561 17844  0 20:58 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
root      8562  8561  0 20:58 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root      8564  8548  0 20:58 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--private.mount.20228
root      8565  8562  0 20:58 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--public.mount.20158
root      8567  8548  0 20:58 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20228,nodev
root      8569  8562  0 20:58 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20158,nodev
root      9268 17844  0 20:59 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root      9269  9268  0 20:59 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root      9281  9269  0 20:59 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20212
root      9284  9269  0 20:59 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20212,nodev
root     11315 17844  0 21:00 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
root     11316 11315  0 21:00 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root     11320 17844  0 21:00 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
root     11321 11320  0 21:00 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root     11323 11316  0 21:00 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--private.mount.20217
root     11331 11321  0 21:00 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--public.mount.20432
root     11333 11316  0 21:00 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20217,nodev
root     11335 11321  0 21:00 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20432,nodev
root     12023 17844  0 21:01 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root     12024 12023  0 21:01 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root     12046 12024  0 21:01 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20317
root     12048 12024  0 21:01 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20317,nodev
root     14080 17844  0 21:02 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
root     14081 14080  0 21:02 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root     14084 17844  0 21:02 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
root     14085 14084  0 21:02 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root     14109 14081  0 21:02 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--private.mount.20052
root     14110 14085  0 21:02 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--public.mount.20064
root     14112 14081  0 21:02 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20052,nodev
root     14115 14085  0 21:02 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20064,nodev
root     14799 17844  0 21:03 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root     14800 14799  0 21:03 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root     14810 14800  0 21:03 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20213
root     14812 14800  0 21:03 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20213,nodev
root     17073 17844  0 21:05 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
root     17074 17073  0 21:05 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root     17076 17844  0 21:05 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
root     17078 17076  0 21:05 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root     17080 17074  0 21:05 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--private.mount.20079
root     17081 17078  0 21:05 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--public.mount.20177
root     17084 17074  0 21:05 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20079,nodev
root     17086 17078  0 21:05 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20177,nodev
root     17787 17844  0 21:05 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root     17789 17787  0 21:05 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root     17791 17789  0 21:05 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20438
root     17793 17789  0 21:05 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20438,nodev
root     17875 17844  0 20:20 ?        00:00:01 python /usr/bin/amazon-efs-mount-watchdog
root     20315 17844  0 21:07 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
root     20316 20315  0 21:07 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root     20319 17844  0 21:07 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
root     20320 20319  0 21:07 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root     20322 20316  0 21:07 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--private.mount.20245
root     20323 20320  0 21:07 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--public.mount.20349
root     20336 20316  0 21:07 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20245,nodev
root     20338 20320  0 21:07 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20349,nodev
root     20795 17844  0 21:07 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
root     20797 20795  0 21:07 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o rw,noexec,nodev,tls
root     20799 20797  0 21:07 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.62515119-fcfb-45ab-a0c0-00daf4c2f9c4.volumes.kubernetes.io~csi.au10025--dev--temp.mount.20225
root     20801 20797  0 21:07 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20225,nodev
root     23746 15284  0 21:09 pts/0    00:00:00 grep mount
root     32204 17844  0 20:52 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--private/mount
root     32205 32204  0 20:52 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--private/mount -o rw,noexec,nodev,tls
root     32238 17844  0 20:52 ?        00:00:00 mount -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--public/mount
root     32239 32238  0 20:52 ?        00:00:00 python /sbin/mount.efs fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--public/mount -o rw,noexec,nodev,tls
root     32307 32205  0 20:52 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.d4382e38-86d8-4264-ac26-367c99f53385.volumes.kubernetes.io~csi.au10025--dev--private.mount.20442
root     32470 32239  0 20:52 ?        00:00:00 stunnel /var/run/efs/stunnel-config.fs-7e4a4047.var.lib.kubelet.pods.d4382e38-86d8-4264-ac26-367c99f53385.volumes.kubernetes.io~csi.au10025--dev--public.mount.20286
root     32473 32205  0 20:52 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--private /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--private/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20442,nodev
root     32477 32239  0 20:52 ?        00:00:00 /sbin/mount.nfs4 127.0.0.1:au10025--dev--public /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--public/mount -o noexec,rw,noresvport,nfsvers=4.1,retrans=2,hard,wsize=1048576,timeo=600,rsize=1048576,port=20286,nodev

The EFS CSI log seems to suggest that the volume just keeps getting mounted, but hangs:

I0308 20:20:57.326496       1 mount_linux.go:163] Cannot run systemd-run, assuming non-systemd OS
I0308 20:20:57.326580       1 mount_linux.go:164] systemd-run failed with: exit status 1
I0308 20:20:57.326601       1 mount_linux.go:165] systemd-run output: Failed to create bus connection: No such file or directory
I0308 20:20:57.326787       1 driver.go:86] Starting watchdog
I0308 20:20:57.327028       1 driver.go:90] Staring subreaper
I0308 20:20:57.327073       1 driver.go:93] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0308 20:20:59.100711       1 node.go:192] NodeGetInfo: called with args
I0308 20:32:02.388637       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:32:02.402424       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:32:02.403443       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047" target_path:"/var/lib/kubelet/pods/072270b2-afa9-453c-9679-cb0683290822/volumes/kubernetes.io~csi/au10025--dev--root/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:32:02.403715       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/072270b2-afa9-453c-9679-cb0683290822/volumes/kubernetes.io~csi/au10025--dev--root/mount
I0308 20:32:02.403738       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:/ at /var/lib/kubelet/pods/072270b2-afa9-453c-9679-cb0683290822/volumes/kubernetes.io~csi/au10025--dev--root/mount with options [tls noexec nodev]
I0308 20:32:02.403754       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:/ /var/lib/kubelet/pods/072270b2-afa9-453c-9679-cb0683290822/volumes/kubernetes.io~csi/au10025--dev--root/mount])
I0308 20:32:04.680968       1 reaper.go:61] Waited for child process 0
I0308 20:32:27.874984       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:32:28.733275       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047" target_path:"/var/lib/kubelet/pods/072270b2-afa9-453c-9679-cb0683290822/volumes/kubernetes.io~csi/au10025--dev--root/mount"
I0308 20:32:28.733631       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/072270b2-afa9-453c-9679-cb0683290822/volumes/kubernetes.io~csi/au10025--dev--root/mount
I0308 20:32:28.733646       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/072270b2-afa9-453c-9679-cb0683290822/volumes/kubernetes.io~csi/au10025--dev--root/mount
I0308 20:32:28.778520       1 reaper.go:61] Waited for child process 0
I0308 20:32:28.836722       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:32:59.253132       1 reaper.go:61] Waited for child process 20
I0308 20:49:55.866156       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:49:55.866511       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:49:55.867955       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:49:55.872004       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:49:55.873074       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:49:55.873172       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:49:55.873192       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:49:55.873205       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 20:49:55.876468       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:49:55.879376       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:49:55.879542       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:49:55.879575       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 20:49:55.879662       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 20:49:56.053428       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:49:56.055128       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:49:56.055392       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:49:56.055509       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 20:49:56.055590       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 20:49:59.016833       1 reaper.go:61] Waited for child process 0
I0308 20:49:59.279839       1 reaper.go:61] Waited for child process 0
I0308 20:49:59.511820       1 reaper.go:61] Waited for child process 0
I0308 20:50:27.880616       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:50:27.889998       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:50:27.890931       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:10.196749       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:10.196764       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:10.196829       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:10.201609       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:10.202728       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:51:10.202827       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:51:10.202849       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 20:51:10.202863       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 20:51:10.203986       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:10.205160       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:51:10.205241       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:51:10.205259       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:51:10.205275       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 20:51:10.357654       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:10.358839       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:51:10.359013       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:51:10.359139       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 20:51:10.359204       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 20:51:13.487189       1 reaper.go:61] Waited for child process 0
I0308 20:51:13.725331       1 reaper.go:61] Waited for child process 0
I0308 20:51:13.776216       1 reaper.go:61] Waited for child process 0
I0308 20:51:27.881033       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:27.881954       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:27.887252       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:52.880855       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--public/mount"
I0308 20:51:52.882856       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--private/mount"
I0308 20:51:52.883225       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:51:52.883237       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:51:52.887385       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:51:52.887397       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:51:52.930456       1 reaper.go:61] Waited for child process 0
I0308 20:51:52.934154       1 reaper.go:61] Waited for child process 0
I0308 20:51:52.935972       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--temp/mount"
I0308 20:51:52.938443       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:51:52.938457       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/5b33ab71-c97b-427b-a874-73d417dc367b/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:51:52.974037       1 reaper.go:61] Waited for child process 0
I0308 20:51:52.983396       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:52.983581       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:52.984535       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:53.352185       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:53.353244       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:51:53.353337       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:51:53.353358       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 20:51:53.353371       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 20:51:53.556968       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:53.561027       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:51:53.561131       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:51:53.561155       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 20:51:53.561170       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 20:51:53.764595       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:51:53.766578       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:51:53.766680       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:51:53.766702       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:51:53.766721       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 20:51:56.505223       1 reaper.go:61] Waited for child process 0
I0308 20:51:56.927840       1 reaper.go:61] Waited for child process 0
I0308 20:51:56.970656       1 reaper.go:61] Waited for child process 0
I0308 20:52:23.726890       1 reaper.go:61] Waited for child process 37
I0308 20:52:23.727111       1 reaper.go:61] Waited for child process 38
I0308 20:52:23.727976       1 reaper.go:61] Waited for child process 40
I0308 20:52:27.877144       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:27.882642       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:27.886322       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:47.791500       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount"
I0308 20:52:47.791942       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:52:47.791953       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:52:47.792174       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--private/mount"
I0308 20:52:47.792638       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:52:47.792647       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:52:47.823288       1 reaper.go:61] Waited for child process 99
E0308 20:52:47.823469       1 driver.go:74] GRPC error: rpc error: code = Internal desc = Could not unmount "/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount": unmount failed: wait: no child processes
Unmounting arguments: /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount
Output:
I0308 20:52:47.829892       1 reaper.go:61] Waited for child process 0
I0308 20:52:47.873358       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--temp/mount"
I0308 20:52:47.876041       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:52:47.876056       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:52:47.889449       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:47.889540       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:47.895876       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:47.896820       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:52:47.897031       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:52:47.897111       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 20:52:47.897190       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 20:52:47.915548       1 reaper.go:61] Waited for child process 0
I0308 20:52:47.983305       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:48.009002       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:48.016939       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:52:48.017570       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:52:48.017680       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 20:52:48.017753       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 20:52:48.206386       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:52:48.207714       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:52:48.207962       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:52:48.208050       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:52:48.208131       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 20:52:51.644291       1 reaper.go:61] Waited for child process 0
I0308 20:53:06.673807       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:53:18.820962       1 reaper.go:61] Waited for child process 59
I0308 20:53:18.821224       1 reaper.go:61] Waited for child process 60
I0308 20:53:18.822105       1 reaper.go:61] Waited for child process 61
I0308 20:53:23.125071       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--temp/mount"
I0308 20:53:23.125391       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:53:23.125403       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:53:23.162851       1 reaper.go:61] Waited for child process 0
I0308 20:53:23.188493       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:53:23.197412       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:53:23.199777       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:53:23.199866       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:53:23.199887       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:53:23.199904       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 20:53:27.886086       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:53:53.875741       1 reaper.go:61] Waited for child process 89
I0308 20:54:15.564803       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:54:15.566216       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:54:32.167006       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:54:47.914063       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--private/mount"
I0308 20:54:47.917256       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:54:47.917281       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:54:47.958414       1 reaper.go:61] Waited for child process 0
I0308 20:54:48.015526       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:54:48.015950       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount"
I0308 20:54:48.015984       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--public/mount"
I0308 20:54:48.016296       1 node.go:154] NodeUnpublishVolume: /var/lib/kubelet/pods/85e7c2f3-9290-4c9d-bb49-8e87e5ab5dd2/volumes/kubernetes.io~csi/au10025--dev--public/mount target not mounted
I0308 20:54:48.018651       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:54:48.018664       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/742d6bd7-3cef-4302-b4f8-fa9420b7d035/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:54:48.021873       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:54:48.022868       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:54:48.023067       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:54:48.023183       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 20:54:48.023275       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 20:54:48.063399       1 reaper.go:61] Waited for child process 0
I0308 20:54:48.115679       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:54:48.123994       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:54:48.125653       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:54:48.125807       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:54:48.125832       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 20:54:48.125849       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 20:55:18.014388       1 reaper.go:61] Waited for child process 87
I0308 20:55:19.024296       1 reaper.go:61] Waited for child process 88
I0308 20:55:23.702429       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:55:23.707798       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:55:23.709003       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:55:23.709075       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:55:23.709098       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:55:23.709114       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 20:56:08.249122       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:56:48.529251       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:56:48.535266       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:56:48.536655       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:56:48.536749       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:56:48.536772       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 20:56:48.536785       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 20:56:48.633559       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:56:48.642285       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:56:48.646217       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:56:48.646345       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:56:48.646367       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 20:56:48.646381       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 20:57:23.747657       1 node.go:134] NodeUnpublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--temp/mount"
I0308 20:57:23.750178       1 node.go:158] NodeUnpublishVolume: unmounting /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:57:23.750193       1 mount_linux.go:202] Unmounting /var/lib/kubelet/pods/d4382e38-86d8-4264-ac26-367c99f53385/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:57:23.783763       1 reaper.go:61] Waited for child process 0
I0308 20:57:23.848243       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:57:23.854120       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:57:23.855175       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:57:23.855426       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:57:23.855556       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:57:23.855661       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 20:57:54.234280       1 reaper.go:61] Waited for child process 116
I0308 20:58:49.625201       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:58:49.630611       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:58:49.631702       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:58:49.631801       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 20:58:49.631822       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 20:58:49.631835       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 20:58:49.725953       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:58:49.733935       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:58:49.735014       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:58:49.735283       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 20:58:49.735408       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 20:58:49.735495       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 20:59:24.397999       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:59:24.408551       1 node.go:176] NodeGetCapabilities: called with args
I0308 20:59:24.410160       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 20:59:24.410247       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 20:59:24.410269       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 20:59:24.410283       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 21:00:51.671716       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:00:51.677227       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:00:51.678637       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:00:51.678732       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 21:00:51.678753       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 21:00:51.678768       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 21:00:51.772250       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:00:51.781345       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:00:51.785665       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:00:51.785780       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 21:00:51.785801       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 21:00:51.785814       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 21:01:25.458392       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:01:25.463851       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:01:25.468215       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:01:25.468315       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 21:01:25.468336       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 21:01:25.468348       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 21:02:55.731427       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:02:55.737283       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:02:55.738599       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:02:55.738694       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 21:02:55.738717       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 21:02:55.738730       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 21:02:55.832573       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:02:55.839808       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:02:55.840672       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:02:55.840817       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 21:02:55.840855       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 21:02:55.840908       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 21:03:27.497782       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:03:27.503835       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:03:27.504770       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:03:27.504873       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 21:03:27.504896       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 21:03:27.504909       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount])




I0308 21:05:03.787558       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:05:03.800166       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:05:03.801235       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:05:03.801328       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 21:05:03.801351       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 21:05:03.801365       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 21:05:03.890703       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:05:03.901556       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:05:03.907914       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:05:03.908014       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 21:05:03.908035       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 21:05:03.908068       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 21:05:31.548115       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:05:31.553794       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:05:31.554971       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:05:31.555069       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 21:05:31.555091       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 21:05:31.555106       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 21:07:19.879652       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:07:19.887250       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:07:19.888436       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:07:19.888536       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 21:07:19.888558       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 21:07:19.888571       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 21:07:19.983208       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:07:19.997229       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:07:19.998769       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:07:19.998955       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 21:07:19.999020       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 21:07:19.999055       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 21:07:39.623624       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:07:39.628823       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:07:39.629732       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:07:39.629825       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 21:07:39.629846       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 21:07:39.629859       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount])
I0308 21:09:51.918497       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:09:51.928076       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:09:51.930807       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--private" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:09:51.930903       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount
I0308 21:09:51.930925       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--private at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount with options [tls noexec nodev]
I0308 21:09:51.930939       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--private /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--private/mount])
I0308 21:09:52.023020       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:09:52.035643       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:09:52.039715       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--public" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:09:52.040684       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
I0308 21:09:52.040724       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--public at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount with options [tls noexec nodev]
I0308 21:09:52.040772       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount])
I0308 21:09:55.633676       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:09:55.640012       1 node.go:176] NodeGetCapabilities: called with args
I0308 21:09:55.641050       1 node.go:50] NodePublishVolume: called with args volume_id:"fs-7e4a4047:au10025--dev--temp" target_path:"/var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount" volume_capability:<mount:<mount_flags:"tls" mount_flags:"noexec" mount_flags:"nodev" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > >
I0308 21:09:55.641153       1 node.go:119] NodePublishVolume: creating dir /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount
I0308 21:09:55.641175       1 node.go:124] NodePublishVolume: mounting fs-7e4a4047:au10025--dev--temp at /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount with options [tls noexec nodev]
I0308 21:09:55.641188       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--temp /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--temp/mount])

I know this isn’t a security group/connectivity issue because this was working fine for a while before it failed, and I have been seen this on other nodes now.

dmesg seems to have some additiona logging information, although I’m not sure how helpful they are:

[Sun Mar  8 21:00:04 2020] INFO: task mount.nfs4:32473 blocked for more than 120 seconds.
[Sun Mar  8 21:00:04 2020]       Not tainted 4.9.0-11-amd64 #1 Debian 4.9.189-3+deb9u1
[Sun Mar  8 21:00:04 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Mar  8 21:00:04 2020] mount.nfs4      D    0 32473  32205 0x00000000
[Sun Mar  8 21:00:04 2020]  0000000000000086 ffff8d6d58069800 0000000000000000 ffff8d6d5638a9c0
[Sun Mar  8 21:00:04 2020]  ffff8d6d6d118980 ffff8d6d5b39e300 ffffa6320732b9c8 ffffffffb6417609
[Sun Mar  8 21:00:04 2020]  ffff8d6d6d3fbcc0 0000000200000000 ffff8d6d6d118980 21174dd02544d08b
[Sun Mar  8 21:00:04 2020] Call Trace:
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417609>] ? __schedule+0x239/0x6f0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417af2>] ? schedule+0x32/0x80
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417daa>] ? schedule_preempt_disabled+0xa/0x10
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6419804>] ? __mutex_lock_slowpath+0xb4/0x130
[Sun Mar  8 21:00:04 2020]  [<ffffffffb641989b>] ? mutex_lock+0x1b/0x30
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0782b04>] ? nfs4_discover_server_trunking+0x44/0x2a0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0786792>] ? nfs_callback_up+0x182/0x470 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078ad10>] ? nfs4_init_client+0x120/0x2a0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc06ef361>] ? __fscache_acquire_cookie+0x61/0x150 [fscache]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc053f38b>] ? __rpc_init_priority_wait_queue+0x7b/0xb0 [sunrpc]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0724575>] ? nfs_get_client+0x2c5/0x3b0 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078a301>] ? nfs4_set_client+0xb1/0x140 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5fa908f>] ? wb_init+0x18f/0x220
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078b7fa>] ? nfs4_create_server+0x12a/0x360 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07834b8>] ? nfs4_remote_mount+0x28/0x50 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb601078b>] ? mount_fs+0x3b/0x160
[Sun Mar  8 21:00:04 2020]  [<ffffffffb602e192>] ? vfs_kern_mount+0x62/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07833ef>] ? nfs_do_root_mount+0x7f/0xc0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078379f>] ? nfs4_try_mount+0x3f/0xc0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07252a1>] ? get_nfs_version+0x21/0x80 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0731f4c>] ? nfs_fs_mount+0x71c/0xda0 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0731680>] ? nfs_clone_super+0x120/0x120 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0730120>] ? param_set_portnr+0x70/0x70 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb601078b>] ? mount_fs+0x3b/0x160
[Sun Mar  8 21:00:04 2020]  [<ffffffffb602e192>] ? vfs_kern_mount+0x62/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffb60306cf>] ? do_mount+0x1cf/0xc60
[Sun Mar  8 21:00:04 2020]  [<ffffffffb603148e>] ? SyS_mount+0x7e/0xd0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5e03b7d>] ? do_syscall_64+0x8d/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffb641c3ce>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[Sun Mar  8 21:00:04 2020] INFO: task mount.nfs4:876 blocked for more than 120 seconds.
[Sun Mar  8 21:00:04 2020]       Not tainted 4.9.0-11-amd64 #1 Debian 4.9.189-3+deb9u1
[Sun Mar  8 21:00:04 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Mar  8 21:00:04 2020] mount.nfs4      D    0   876    807 0x00000000
[Sun Mar  8 21:00:04 2020]  0000000000000086 ffff8d6cff88a000 0000000000000000 ffff8d6c70680100
[Sun Mar  8 21:00:04 2020]  ffff8d6d6d118980 ffff8d6d5b39e300 ffffa632083639c8 ffffffffb6417609
[Sun Mar  8 21:00:04 2020]  ffff8d6d6d3fbcc0 0000000200000000 ffff8d6d6d118980 ffffffffb5ec52ee
[Sun Mar  8 21:00:04 2020] Call Trace:
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417609>] ? __schedule+0x239/0x6f0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5ec52ee>] ? mutex_optimistic_spin+0xee/0x1c0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417af2>] ? schedule+0x32/0x80
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417daa>] ? schedule_preempt_disabled+0xa/0x10
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6419804>] ? __mutex_lock_slowpath+0xb4/0x130
[Sun Mar  8 21:00:04 2020]  [<ffffffffb641989b>] ? mutex_lock+0x1b/0x30
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0782b04>] ? nfs4_discover_server_trunking+0x44/0x2a0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0786792>] ? nfs_callback_up+0x182/0x470 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078ad10>] ? nfs4_init_client+0x120/0x2a0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc06ef361>] ? __fscache_acquire_cookie+0x61/0x150 [fscache]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc053f38b>] ? __rpc_init_priority_wait_queue+0x7b/0xb0 [sunrpc]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0724575>] ? nfs_get_client+0x2c5/0x3b0 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078a301>] ? nfs4_set_client+0xb1/0x140 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5fa908f>] ? wb_init+0x18f/0x220
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078b7fa>] ? nfs4_create_server+0x12a/0x360 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07834b8>] ? nfs4_remote_mount+0x28/0x50 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb601078b>] ? mount_fs+0x3b/0x160
[Sun Mar  8 21:00:04 2020]  [<ffffffffb602e192>] ? vfs_kern_mount+0x62/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07833ef>] ? nfs_do_root_mount+0x7f/0xc0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078379f>] ? nfs4_try_mount+0x3f/0xc0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07252a1>] ? get_nfs_version+0x21/0x80 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0731f4c>] ? nfs_fs_mount+0x71c/0xda0 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0731680>] ? nfs_clone_super+0x120/0x120 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0730120>] ? param_set_portnr+0x70/0x70 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb601078b>] ? mount_fs+0x3b/0x160
[Sun Mar  8 21:00:04 2020]  [<ffffffffb602e192>] ? vfs_kern_mount+0x62/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffb60306cf>] ? do_mount+0x1cf/0xc60
[Sun Mar  8 21:00:04 2020]  [<ffffffffb603148e>] ? SyS_mount+0x7e/0xd0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5e03b7d>] ? do_syscall_64+0x8d/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffb641c3ce>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[Sun Mar  8 21:00:04 2020] INFO: task mount.nfs4:3033 blocked for more than 120 seconds.
[Sun Mar  8 21:00:04 2020]       Not tainted 4.9.0-11-amd64 #1 Debian 4.9.189-3+deb9u1
[Sun Mar  8 21:00:04 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Sun Mar  8 21:00:04 2020] mount.nfs4      D    0  3033   2822 0x00000000
[Sun Mar  8 21:00:04 2020]  0000000000000086 ffff8d6cf6c17400 ffff8d6cf6c17000 ffff8d6d293b0780
[Sun Mar  8 21:00:04 2020]  ffff8d6d6d118980 ffff8d6c70628480 ffffa6320433b9c8 ffffffffb6417609
[Sun Mar  8 21:00:04 2020]  ffff8d6d6d3fbcc0 0000000200000000 ffff8d6d6d118980 ffffffffb5ec52ee
[Sun Mar  8 21:00:04 2020] Call Trace:
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417609>] ? __schedule+0x239/0x6f0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5ec52ee>] ? mutex_optimistic_spin+0xee/0x1c0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417af2>] ? schedule+0x32/0x80
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6417daa>] ? schedule_preempt_disabled+0xa/0x10
[Sun Mar  8 21:00:04 2020]  [<ffffffffb6419804>] ? __mutex_lock_slowpath+0xb4/0x130
[Sun Mar  8 21:00:04 2020]  [<ffffffffb641989b>] ? mutex_lock+0x1b/0x30
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0782b04>] ? nfs4_discover_server_trunking+0x44/0x2a0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0786792>] ? nfs_callback_up+0x182/0x470 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078ad10>] ? nfs4_init_client+0x120/0x2a0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc06ef361>] ? __fscache_acquire_cookie+0x61/0x150 [fscache]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc053f38b>] ? __rpc_init_priority_wait_queue+0x7b/0xb0 [sunrpc]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0724575>] ? nfs_get_client+0x2c5/0x3b0 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078a301>] ? nfs4_set_client+0xb1/0x140 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5fa908f>] ? wb_init+0x18f/0x220
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078b7fa>] ? nfs4_create_server+0x12a/0x360 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07834b8>] ? nfs4_remote_mount+0x28/0x50 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb601078b>] ? mount_fs+0x3b/0x160
[Sun Mar  8 21:00:04 2020]  [<ffffffffb602e192>] ? vfs_kern_mount+0x62/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07833ef>] ? nfs_do_root_mount+0x7f/0xc0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc078379f>] ? nfs4_try_mount+0x3f/0xc0 [nfsv4]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc07252a1>] ? get_nfs_version+0x21/0x80 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0731f4c>] ? nfs_fs_mount+0x71c/0xda0 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0731680>] ? nfs_clone_super+0x120/0x120 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffc0730120>] ? param_set_portnr+0x70/0x70 [nfs]
[Sun Mar  8 21:00:04 2020]  [<ffffffffb601078b>] ? mount_fs+0x3b/0x160
[Sun Mar  8 21:00:04 2020]  [<ffffffffb602e192>] ? vfs_kern_mount+0x62/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffb60306cf>] ? do_mount+0x1cf/0xc60
[Sun Mar  8 21:00:04 2020]  [<ffffffffb603148e>] ? SyS_mount+0x7e/0xd0
[Sun Mar  8 21:00:04 2020]  [<ffffffffb5e03b7d>] ? do_syscall_64+0x8d/0x100
[Sun Mar  8 21:00:04 2020]  [<ffffffffb641c3ce>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6

messages like the above are repeated, seemingly once for each failed mount attempt.

What you expected to happen? For the volumes to mount fine, every time 😃

How to reproduce it (as minimally and precisely as possible)? As much as I would love to give you some reproduction instructions since it would speed up a fix, unfortunately I haven’t found the magic formula for what exactly causes this problem.

This seems very similar if not exactly the same as the now closed #115, however I had previously seem this in our production clusters (which are running but it seemed fixed after the introduction of watchdog).

I think in reality this problem was never really fixed but is just very elusive, or perhaps it’s a regression in 0.3.0?

Anything else we need to know?: If I try to recover from this issue by killing the hanging mount commands, I see output like this in the logs (not sure if it’s helpful):

E0308 21:15:49.693070       1 driver.go:74] GRPC error: rpc error: code = Internal desc = Could not mount "fs-7e4a4047:au10025--dev--public" at "/var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount": mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/2d954c71-b531-4c30-929a-085e636c63c3/volumes/kubernetes.io~csi/au10025--dev--public/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
Failed to initialize TLS tunnel for fs-7e4a4047
I0308 21:15:49.693118       1 reaper.go:61] Waited for child process 0
I0308 21:15:49.707751       1 reaper.go:61] Waited for child process 0
E0308 21:15:49.707748       1 mount_linux.go:140] Mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t efs -o tls,noexec,nodev fs-7e4a4047:au10025--dev--public /var/lib/kubelet/pods/62515119-fcfb-45ab-a0c0-00daf4c2f9c4/volumes/kubernetes.io~csi/au10025--dev--public/mount
Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
Failed to initialize TLS tunnel for fs-7e4a4047

However when the affected pod restarts, it is able to successfully mount the volumes again for some time, before the problem eventually reappears.

Environment

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:29Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

Hosts run debian provisioned by kops

• Driver version: 0.3.0