ceph-csi: csi-cephfsplugin After the plug-in service is restarted, the client service cannot mount the storage disk

Environmental inventory： kubernetes kubeadm v1.17 docker: 19.03.4 ceph: 14.2.11 csi: v3.1.2

Problem phenomenon After the CSI cephfsplugin service is restarted, the client has the following problems. You need to restart the client service container to return to normal

root@nginx-cephfs1-77664d8bfb-mpbwp:/# df -hT
df: /usr/share/nginx/html: Transport endpoint is not connected
Filesystem              Type     Size  Used Avail Use% Mounted on
overlay                 overlay  112G   31G   81G  28% /
tmpfs                   tmpfs     64M     0   64M   0% /dev
tmpfs                   tmpfs    7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/mapper/centos-home xfs      112G   31G   81G  28% /root
shm                     tmpfs     64M     0   64M   0% /dev/shm
tmpfs                   tmpfs    7.9G   12K  7.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                   tmpfs    7.9G     0  7.9G   0% /proc/acpi
tmpfs                   tmpfs    7.9G     0  7.9G   0% /proc/scsi
tmpfs                   tmpfs    7.9G     0  7.9G   0% /sys/firmware

csi-cephfsplugin:

I1231 06:15:10.002371       1 utils.go:160] ID: 32 GRPC request: {}
I1231 06:15:10.006170       1 utils.go:165] ID: 32 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
I1231 06:15:10.008460       1 utils.go:159] ID: 33 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1231 06:15:10.010427       1 utils.go:160] ID: 33 GRPC request: {"volume_id":"0001-0024-9771f5aa-191c-4e16-9524-19dc31d2cd8d-0000000000000001-fa920d17-4b2d-11eb-be5f-bacc5498d6bd","volume_path":"/home/kubelet/pods/e05631d7-e406-4799-b7a5-dc8819439dd1/volumes/kubernetes.io~csi/pvc-1f3ed29b-2709-4ce1-9dfc-c8b201c6711b/mount"}
E1231 06:15:10.023137       1 utils.go:163] ID: 33 GRPC error: rpc error: code = Internal desc = stat /home/kubelet/pods/e05631d7-e406-4799-b7a5-dc8819439dd1/volumes/kubernetes.io~csi/pvc-1f3ed29b-2709-4ce1-9dfc-c8b201c6711b/mount: transport endpoint is not connected
I1231 06:15:15.748857       1 utils.go:159] ID: 34 GRPC call: /csi.v1.Node/NodeGetCapabilities
I1231 06:15:15.750801       1 utils.go:160] ID: 34 GRPC request: {}
I1231 06:15:15.755824       1 utils.go:165] ID: 34 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
I1231 06:15:15.757638       1 utils.go:159] ID: 35 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1231 06:15:15.758772       1 utils.go:160] ID: 35 GRPC request: {"volume_id":"0001-0024-9771f5aa-191c-4e16-9524-19dc31d2cd8d-0000000000000001-52fd8606-40fd-11eb-8ffe-eaf01d57385a","volume_path":"/home/kubelet/pods/ae344b80-3b07-4589-b1a1-ca75fa9debf2/volumes/kubernetes.io~csi/pvc-ec69de59-7823-4840-8eee-544f8261fef0/mount"}
E1231 06:15:15.767358       1 utils.go:163] ID: 35 GRPC error: rpc error: code = Internal desc = stat /home/kubelet/pods/ae344b80-3b07-4589-b1a1-ca75fa9debf2/volumes/kubernetes.io~csi/pvc-ec69de59-7823-4840-8eee-544f8261fef0/mount: transport endpoint is not connected
I1231 06:16:03.804767       1 utils.go:159] ID: 36 GRPC call: /csi.v1.Identity/Probe
I1231 06:16:03.805863       1 utils.go:160] ID: 36 GRPC request: {}
I1231 06:16:03.807203       1 utils.go:165] ID: 36 GRPC response: {}
I1231 06:16:43.208991       1 utils.go:159] ID: 37 GRPC call: /csi.v1.Node/NodeGetCapabilities
I1231 06:16:43.210732       1 utils.go:160] ID: 37 GRPC request: {}
I1231 06:16:43.214016       1 utils.go:165] ID: 37 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
I1231 06:16:43.225537       1 utils.go:159] ID: 38 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1231 06:16:43.227063       1 utils.go:160] ID: 38 GRPC request: {"volume_id":"0001-0024-9771f5aa-191c-4e16-9524-19dc31d2cd8d-0000000000000001-52fd8606-40fd-11eb-8ffe-eaf01d57385a","volume_path":"/home/kubelet/pods/ae344b80-3b07-4589-b1a1-ca75fa9debf2/volumes/kubernetes.io~csi/pvc-ec69de59-7823-4840-8eee-544f8261fef0/mount"}
E1231 06:16:43.239974       1 utils.go:163] ID: 38 GRPC error: rpc error: code = Internal desc = stat /home/kubelet/pods/ae344b80-3b07-4589-b1a1-ca75fa9debf2/volumes/kubernetes.io~csi/pvc-ec69de59-7823-4840-8eee-544f8261fef0/mount: transport endpoint is not connected
I1231 06:17:03.805346       1 utils.go:159] ID: 39 GRPC call: /csi.v1.Identity/Probe
I1231 06:17:03.807484       1 utils.go:160] ID: 39 GRPC request: {}
I1231 06:17:03.808654       1 utils.go:165] ID: 39 GRPC response: {}
I1231 06:17:03.988552       1 utils.go:159] ID: 40 GRPC call: /csi.v1.Node/NodeGetCapabilities
I1231 06:17:03.989213       1 utils.go:160] ID: 40 GRPC request: {}
I1231 06:17:03.990647       1 utils.go:165] ID: 40 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
I1231 06:17:03.992281       1 utils.go:159] ID: 41 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1231 06:17:03.993917       1 utils.go:160] ID: 41 GRPC request: {"volume_id":"0001-0024-9771f5aa-191c-4e16-9524-19dc31d2cd8d-0000000000000001-fa920d17-4b2d-11eb-be5f-bacc5498d6bd","volume_path":"/home/kubelet/pods/e05631d7-e406-4799-b7a5-dc8819439dd1/volumes/kubernetes.io~csi/pvc-1f3ed29b-2709-4ce1-9dfc-c8b201c6711b/mount"}
E1231 06:17:04.009064       1 utils.go:163] ID: 41 GRPC error: rpc error: code = Internal desc = stat /home/kubelet/pods/e05631d7-e406-4799-b7a5-dc8819439dd1/volumes/kubernetes.io~csi/pvc-1f3ed29b-2709-4ce1-9dfc-c8b201c6711b/mount: transport endpoint is not connected
I1231 06:17:53.322064       1 utils.go:159] ID: 42 GRPC call: /csi.v1.Node/NodeGetCapabilities
I1231 06:17:53.327475       1 utils.go:160] ID: 42 GRPC request: {}
I1231 06:17:53.331634       1 utils.go:165] ID: 42 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
I1231 06:17:53.334161       1 utils.go:159] ID: 43 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1231 06:17:53.335814       1 utils.go:160] ID: 43 GRPC request: {"volume_id":"0001-0024-9771f5aa-191c-4e16-9524-19dc31d2cd8d-0000000000000001-52fd8606-40fd-11eb-8ffe-eaf01d57385a","volume_path":"/home/kubelet/pods/ae344b80-3b07-4589-b1a1-ca75fa9debf2/volumes/kubernetes.io~csi/pvc-ec69de59-7823-4840-8eee-544f8261fef0/mount"}
E1231 06:17:53.345216       1 utils.go:163] ID: 43 GRPC error: rpc error: code = Internal desc = stat /home/kubelet/pods/ae344b80-3b07-4589-b1a1-ca75fa9debf2/volumes/kubernetes.io~csi/pvc-ec69de59-7823-4840-8eee-544f8261fef0/mount: transport endpoint is not connected
I1231 06:18:03.804955       1 utils.go:159] ID: 44 GRPC call: /csi.v1.Identity/Probe
I1231 06:18:03.806929       1 utils.go:160] ID: 44 GRPC request: {}
I1231 06:18:03.808853       1 utils.go:165] ID: 44 GRPC response: {}
I1231 06:18:18.573703       1 utils.go:159] ID: 45 GRPC call: /csi.v1.Node/NodeGetCapabilities
I1231 06:18:18.575350       1 utils.go:160] ID: 45 GRPC request: {}
I1231 06:18:18.583726       1 utils.go:165] ID: 45 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}}]}
I1231 06:18:18.585569       1 utils.go:159] ID: 46 GRPC call: /csi.v1.Node/NodeGetVolumeStats
I1231 06:18:18.586131       1 utils.go:160] ID: 46 GRPC request: {"volume_id":"0001-0024-9771f5aa-191c-4e16-9524-19dc31d2cd8d-0000000000000001-fa920d17-4b2d-11eb-be5f-bacc5498d6bd","volume_path":"/home/kubelet/pods/e05631d7-e406-4799-b7a5-dc8819439dd1/volumes/kubernetes.io~csi/pvc-1f3ed29b-2709-4ce1-9dfc-c8b201c6711b/mount"}
E1231 06:18:18.599702       1 utils.go:163] ID: 46 GRPC error: rpc error: code = Internal desc = stat /home/kubelet/pods/e05631d7-e406-4799-b7a5-dc8819439dd1/volumes/kubernetes.io~csi/pvc-1f3ed29b-2709-4ce1-9dfc-c8b201c6711b/mount: transport endpoint is not connected
I1231 06:19:03.805562       1 utils.go:159] ID: 47 GRPC call: /csi.v1.Identity/Probe
I1231 06:19:03.807591       1 utils.go:160] ID: 47 GRPC request: {}
I1231 06:19:03.809026       1 utils.go:165] ID: 47 GRPC response: {}

If you need other service logs, please contact me. Thank you for your help