longhorn: I get error on csi-attacher, csi-provisioner, csi-resizer and csi-snapshotter in specific time period.

Question Some of my longhorn workloads (csi-attacher, csi-provisioner, csi-resizer and csi-snapshotter) have errors at specific time through the day. (through 11:00 PM To 12:40 AM)

Environment:

• Longhorn version: I have errors on both longhorn versions (1.0.2 and 1.1.0)
• Kubernetes version: 1.17.11
• Node config
  • OS type and version: debian 10 buster
  • CPU per node: 4VCPU
  • Memory per node: 4GB
  • Disk type: Ceph on cloud: (My VMs are in openstack cloud and their storage are ceph)

Additional context My workloads which have errors:

csi-provisioner log: I0227 20:11:09.483413 1 feature_gate.go:243] feature gates: &{map[]} I0227 20:11:09.483544 1 csi-provisioner.go:107] Version: v1.6.0-lh1-0-g7fee42b60 I0227 20:11:09.483614 1 csi-provisioner.go:121] Building kube configs for running in cluster… I0227 20:11:09.492373 1 connection.go:153] Connecting to unix:///csi/csi.sock I0227 20:11:09.494139 1 common.go:111] Probing CSI driver for readiness I0227 20:11:09.494192 1 connection.go:182] GRPC call: /csi.v1.Identity/Probe I0227 20:11:09.494200 1 connection.go:183] GRPC request: {} I0227 20:11:09.502784 1 connection.go:185] GRPC response: {} I0227 20:11:09.503194 1 connection.go:186] GRPC error: <nil> I0227 20:11:09.503204 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo I0227 20:11:09.503207 1 connection.go:183] GRPC request: {} I0227 20:11:09.505204 1 connection.go:185] GRPC response: {“name”:“driver.longhorn.io”,“vendor_version”:“v1.1.0”} I0227 20:11:09.505528 1 connection.go:186] GRPC error: <nil> I0227 20:11:09.505538 1 csi-provisioner.go:163] Detected CSI driver driver.longhorn.io W0227 20:11:09.505564 1 metrics.go:142] metrics endpoint will not be started because metrics-address was not specified. I0227 20:11:09.505571 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities I0227 20:11:09.505574 1 connection.go:183] GRPC request: {} I0227 20:11:09.508620 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Service”:{“type”:1}}},{“Type”:{“VolumeExpansion”:{“type”:2}}}]} I0227 20:11:09.510008 1 connection.go:186] GRPC error: <nil> I0227 20:11:09.510022 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0227 20:11:09.510026 1 connection.go:183] GRPC request: {} I0227 20:11:09.515286 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Rpc”:{“type”:1}}},{“Type”:{“Rpc”:{“type”:2}}},{“Type”:{“Rpc”:{“type”:9}}},{“Type”:{“Rpc”:{“type”:5}}}]} I0227 20:11:09.517721 1 connection.go:186] GRPC error: <nil> I0227 20:11:09.518200 1 controller.go:709] Using saving PVs to API server in background I0227 20:11:09.518458 1 leaderelection.go:242] attempting to acquire leader lease longhorn-system/driver-longhorn-io… I0227 20:11:10.915796 1 leaderelection.go:352] lock is held by csi-provisioner-7645579485-ntmfk and has not yet expired I0227 20:11:10.915882 1 leaderelection.go:247] failed to acquire lease longhorn-system/driver-longhorn-io I0227 20:11:10.915912 1 leader_election.go:172] new leader detected, current leader: csi-provisioner-7645579485-ntmfk E0227 20:11:42.264649 1 leaderelection.go:331] error retrieving resource lock longhorn-system/driver-longhorn-io: etcdserver: request timed out I0227 20:11:42.264692 1 leaderelection.go:247] failed to acquire lease longhorn-system/driver-longhorn-io I0227 20:11:52.428806 1 leaderelection.go:352] lock is held by csi-provisioner-7645579485-ntmfk and has not yet expired I0227 20:11:52.428828 1 leaderelection.go:247] failed to acquire lease longhorn-system/driver-longhorn-io I0227 20:12:02.724822 1 leaderelection.go:352] lock is held by csi-provisioner-7645579485-ntmfk and has not yet expired I0227 20:12:02.724854 1 leaderelection.go:247] failed to acquire lease longhorn-system/driver-longhorn-io I0227 20:12:27.250296 1 leaderelection.go:252] successfully acquired lease longhorn-system/driver-longhorn-io I0227 20:12:27.250306 1 leader_election.go:172] new leader detected, current leader: csi-provisioner-7645579485-rshxx I0227 20:12:27.250609 1 leader_election.go:165] became leader, starting I0227 20:12:27.250983 1 reflector.go:153] Starting reflector *v1.StorageClass (1h0m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.251012 1 reflector.go:188] Listing and watching *v1.StorageClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.250993 1 reflector.go:153] Starting reflector *v1.PersistentVolumeClaim (15m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.251056 1 reflector.go:188] Listing and watching *v1.PersistentVolumeClaim from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.350989 1 shared_informer.go:227] caches populated I0227 20:12:27.351029 1 shared_informer.go:227] caches populated I0227 20:12:27.351261 1 controller.go:799] Starting provisioner controller driver.longhorn.io_csi-provisioner-7645579485-rshxx_2a536b4c-a212-4a0d-ab54-50a681f36fa0! I0227 20:12:27.351389 1 reflector.go:153] Starting reflector *v1.StorageClass (15m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.351410 1 reflector.go:188] Listing and watching *v1.StorageClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.351492 1 reflector.go:153] Starting reflector *v1.PersistentVolume (15m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.351521 1 reflector.go:188] Listing and watching *v1.PersistentVolume from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.358767 1 volume_store.go:97] Starting save volume queue I0227 20:12:27.358778 1 clone_controller.go:58] Starting CloningProtection controller I0227 20:12:27.358838 1 clone_controller.go:74] Started CloningProtection controller I0227 20:12:27.358879 1 reflector.go:153] Starting reflector *v1.PersistentVolumeClaim (15m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.358888 1 reflector.go:188] Listing and watching *v1.PersistentVolumeClaim from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:12:27.451673 1 shared_informer.go:227] caches populated I0227 20:12:27.452231 1 controller.go:848] Started provisioner controller driver.longhorn.io_csi-provisioner-7645579485-rshxx_2a536b4c-a212-4a0d-ab54-50a681f36fa0! I0227 20:12:37.252151 1 leaderelection.go:288] failed to renew lease longhorn-system/driver-longhorn-io: failed to tryAcquireOrRenew context deadline exceeded F0227 20:12:37.252254 1 leader_election.go:169] stopped leading Wrap lines Previous Container

csi-attacher logs:

I0227 20:13:20.162039 1 main.go:91] Version: v2.2.1-lh1-0-gda3ee62d I0227 20:13:20.163597 1 connection.go:153] Connecting to unix:///csi/csi.sock I0227 20:13:20.164728 1 common.go:111] Probing CSI driver for readiness I0227 20:13:20.164757 1 connection.go:182] GRPC call: /csi.v1.Identity/Probe I0227 20:13:20.164763 1 connection.go:183] GRPC request: {} I0227 20:13:20.175837 1 connection.go:185] GRPC response: {} I0227 20:13:20.176856 1 connection.go:186] GRPC error: <nil> I0227 20:13:20.176898 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo I0227 20:13:20.176916 1 connection.go:183] GRPC request: {} I0227 20:13:20.181971 1 connection.go:185] GRPC response: {“name”:“driver.longhorn.io”,“vendor_version”:“v1.1.0”} I0227 20:13:20.183455 1 connection.go:186] GRPC error: <nil> I0227 20:13:20.183487 1 main.go:136] CSI driver name: “driver.longhorn.io” W0227 20:13:20.183540 1 metrics.go:146] metrics endpoint will not be started because metrics-address was not specified. I0227 20:13:20.183570 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities I0227 20:13:20.183592 1 connection.go:183] GRPC request: {} I0227 20:13:20.187280 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Service”:{“type”:1}}},{“Type”:{“VolumeExpansion”:{“type”:2}}}]} I0227 20:13:20.190780 1 connection.go:186] GRPC error: <nil> I0227 20:13:20.190802 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0227 20:13:20.190817 1 connection.go:183] GRPC request: {} I0227 20:13:20.200711 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Rpc”:{“type”:1}}},{“Type”:{“Rpc”:{“type”:2}}},{“Type”:{“Rpc”:{“type”:9}}},{“Type”:{“Rpc”:{“type”:5}}}]} I0227 20:13:20.202516 1 connection.go:186] GRPC error: <nil> I0227 20:13:20.202618 1 main.go:162] CSI driver supports ControllerPublishUnpublish, using real CSI handler I0227 20:13:20.202631 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0227 20:13:20.202635 1 connection.go:183] GRPC request: {} I0227 20:13:20.205107 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Rpc”:{“type”:1}}},{“Type”:{“Rpc”:{“type”:2}}},{“Type”:{“Rpc”:{“type”:9}}},{“Type”:{“Rpc”:{“type”:5}}}]} I0227 20:13:20.208734 1 connection.go:186] GRPC error: <nil> I0227 20:13:20.209119 1 leaderelection.go:242] attempting to acquire leader lease longhorn-system/external-attacher-leader-driver-longhorn-io… E0227 20:13:37.022063 1 leaderelection.go:331] error retrieving resource lock longhorn-system/external-attacher-leader-driver-longhorn-io: etcdserver: leader changed I0227 20:13:37.022100 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:13:52.030654 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:13:52.030690 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:13:52.030732 1 leader_election.go:172] new leader detected, current leader: csi-attacher-8c6788bc-lbpr2 I0227 20:14:02.678795 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:14:02.678825 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:14:12.030075 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:14:12.030092 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:14:20.133726 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:14:20.133751 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:14:27.707138 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:14:27.707167 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:14:37.072457 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:14:37.072488 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io E0227 20:15:02.314812 1 leaderelection.go:331] error retrieving resource lock longhorn-system/external-attacher-leader-driver-longhorn-io: etcdserver: request timed out I0227 20:15:02.314866 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io E0227 20:15:23.321951 1 leaderelection.go:331] error retrieving resource lock longhorn-system/external-attacher-leader-driver-longhorn-io: etcdserver: request timed out I0227 20:15:23.321970 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:15:28.909776 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:15:28.909820 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:15:35.724467 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:15:35.724498 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:15:43.873232 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:15:43.873252 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:15:53.975808 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:15:53.975832 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:00.269198 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:16:00.269238 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:07.565911 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:16:07.565941 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:15.010733 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:16:15.010794 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:23.881920 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:16:23.881975 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:38.295732 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:16:38.295768 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:47.916464 1 leaderelection.go:352] lock is held by csi-attacher-8c6788bc-lbpr2 and has not yet expired I0227 20:16:47.916500 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:57.027430 1 leaderelection.go:252] successfully acquired lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:16:57.027516 1 leader_election.go:172] new leader detected, current leader: csi-attacher-8c6788bc-7bzcn I0227 20:16:57.027569 1 leader_election.go:165] became leader, starting I0227 20:16:57.027716 1 controller.go:121] Starting CSI attacher I0227 20:16:57.028038 1 reflector.go:153] Starting reflector *v1beta1.CSINode (10m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:57.028090 1 reflector.go:188] Listing and watching *v1beta1.CSINode from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:57.028040 1 reflector.go:153] Starting reflector *v1beta1.VolumeAttachment (10m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:57.028123 1 reflector.go:188] Listing and watching *v1beta1.VolumeAttachment from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:57.028045 1 reflector.go:153] Starting reflector *v1.PersistentVolume (10m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:57.028461 1 reflector.go:188] Listing and watching *v1.PersistentVolume from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:57.127984 1 shared_informer.go:227] caches populated I0227 20:16:57.128117 1 controller.go:232] Started PV processing “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” I0227 20:16:57.128133 1 controller.go:198] Started VA processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:57.128179 1 csi_handler.go:209] CSIHandler: processing VA “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:57.128194 1 csi_handler.go:236] Attaching “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:57.128213 1 csi_handler.go:591] CSIHandler: processing PV “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” I0227 20:16:57.128436 1 csi_handler.go:627] CSIHandler: processing PV “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07”: VA “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” found I0227 20:16:57.128219 1 csi_handler.go:396] Starting attach operation for “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:57.128514 1 csi_handler.go:559] Saving attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:57.262085 1 csi_handler.go:570] Saved attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:57.262123 1 csi_handler.go:219] Error processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7”: failed to attach: PersistentVolume “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” is marked for deletion I0227 20:16:58.262333 1 controller.go:198] Started VA processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:58.262368 1 csi_handler.go:209] CSIHandler: processing VA “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:58.262377 1 csi_handler.go:236] Attaching “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:58.262386 1 csi_handler.go:396] Starting attach operation for “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:58.262398 1 csi_handler.go:559] Saving attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:58.435776 1 controller.go:158] Ignoring VolumeAttachment “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” change I0227 20:16:59.650510 1 controller.go:158] Ignoring VolumeAttachment “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” change I0227 20:16:59.653714 1 csi_handler.go:570] Saved attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:16:59.653734 1 csi_handler.go:219] Error processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7”: failed to attach: PersistentVolume “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” is marked for deletion I0227 20:17:00.685937 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:01.653893 1 controller.go:198] Started VA processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:01.653932 1 csi_handler.go:209] CSIHandler: processing VA “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:01.653940 1 csi_handler.go:236] Attaching “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:01.653947 1 csi_handler.go:396] Starting attach operation for “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:01.653958 1 csi_handler.go:559] Saving attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:03.927942 1 controller.go:158] Ignoring VolumeAttachment “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” change I0227 20:17:03.972960 1 csi_handler.go:570] Saved attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:03.972994 1 csi_handler.go:219] Error processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7”: failed to attach: PersistentVolume “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” is marked for deletion I0227 20:17:06.112880 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:07.973328 1 controller.go:198] Started VA processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:07.973366 1 csi_handler.go:209] CSIHandler: processing VA “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:07.973375 1 csi_handler.go:236] Attaching “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:07.973385 1 csi_handler.go:396] Starting attach operation for “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:07.973398 1 csi_handler.go:559] Saving attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:08.033439 1 csi_handler.go:570] Saved attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:08.033493 1 csi_handler.go:219] Error processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7”: failed to attach: PersistentVolume “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” is marked for deletion I0227 20:17:08.266587 1 controller.go:158] Ignoring VolumeAttachment “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” change I0227 20:17:11.444817 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:16.033800 1 controller.go:198] Started VA processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:16.033893 1 csi_handler.go:209] CSIHandler: processing VA “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:16.033912 1 csi_handler.go:236] Attaching “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:16.033928 1 csi_handler.go:396] Starting attach operation for “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:16.033970 1 csi_handler.go:559] Saving attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:16.130061 1 csi_handler.go:570] Saved attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:16.130143 1 csi_handler.go:219] Error processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7”: failed to attach: PersistentVolume “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” is marked for deletion I0227 20:17:16.130902 1 controller.go:158] Ignoring VolumeAttachment “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” change I0227 20:17:18.107202 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:24.166690 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:30.705955 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:32.130371 1 controller.go:198] Started VA processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:32.130429 1 csi_handler.go:209] CSIHandler: processing VA “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:32.130447 1 csi_handler.go:236] Attaching “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:32.130457 1 csi_handler.go:396] Starting attach operation for “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:32.130483 1 csi_handler.go:559] Saving attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:32.535305 1 csi_handler.go:570] Saved attach error to “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” I0227 20:17:32.535527 1 csi_handler.go:219] Error processing “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7”: failed to attach: PersistentVolume “pvc-cbc9f195-8e4c-4242-b36c-b132531c5d07” is marked for deletion I0227 20:17:34.884552 1 controller.go:158] Ignoring VolumeAttachment “csi-4e0daa4b82e883dd1a8fea9f60028d88c68caa898d6c4d9f577157c13bf8d1f7” change I0227 20:17:35.771846 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:41.596901 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-attacher-leader-driver-longhorn-io I0227 20:17:56.597548 1 leaderelection.go:288] failed to renew lease longhorn-system/external-attacher-leader-driver-longhorn-io: failed to tryAcquireOrRenew context deadline exceeded F0227 20:17:56.597601 1 leader_election.go:169] stopped leading Wrap lines Previous Container

csi-resizer logs: I0227 20:14:00.232811 1 main.go:68] Version : v0.5.1-lh1-0-g1094448 I0227 20:14:00.235795 1 connection.go:153] Connecting to unix:///csi/csi.sock I0227 20:14:00.237037 1 common.go:111] Probing CSI driver for readiness I0227 20:14:00.237103 1 connection.go:182] GRPC call: /csi.v1.Identity/Probe I0227 20:14:00.237118 1 connection.go:183] GRPC request: {} I0227 20:14:00.246152 1 connection.go:185] GRPC response: {} I0227 20:14:00.246867 1 connection.go:186] GRPC error: <nil> I0227 20:14:00.246894 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo I0227 20:14:00.246901 1 connection.go:183] GRPC request: {} I0227 20:14:00.249868 1 connection.go:185] GRPC response: {“name”:“driver.longhorn.io”,“vendor_version”:“v1.1.0”} I0227 20:14:00.250772 1 connection.go:186] GRPC error: <nil> I0227 20:14:00.250866 1 csi_resizer.go:79] CSI driver name: “driver.longhorn.io” W0227 20:14:00.250968 1 metrics.go:146] metrics endpoint will not be started because metrics-address was not specified. I0227 20:14:00.251048 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities I0227 20:14:00.251128 1 connection.go:183] GRPC request: {} I0227 20:14:00.257176 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Service”:{“type”:1}}},{“Type”:{“VolumeExpansion”:{“type”:2}}}]} I0227 20:14:00.258408 1 connection.go:186] GRPC error: <nil> I0227 20:14:00.258431 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0227 20:14:00.258435 1 connection.go:183] GRPC request: {} I0227 20:14:00.262717 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Rpc”:{“type”:1}}},{“Type”:{“Rpc”:{“type”:2}}},{“Type”:{“Rpc”:{“type”:9}}},{“Type”:{“Rpc”:{“type”:5}}}]} I0227 20:14:00.265176 1 connection.go:186] GRPC error: <nil> I0227 20:14:00.265587 1 leaderelection.go:242] attempting to acquire leader lease longhorn-system/external-resizer-driver-longhorn-io… I0227 20:14:00.579644 1 leaderelection.go:352] lock is held by csi-resizer-7b5b48db77-78r8c and has not yet expired I0227 20:14:00.579706 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-resizer-driver-longhorn-io I0227 20:14:00.579750 1 leader_election.go:172] new leader detected, current leader: csi-resizer-7b5b48db77-78r8c I0227 20:14:09.432683 1 leaderelection.go:352] lock is held by csi-resizer-7b5b48db77-78r8c and has not yet expired I0227 20:14:09.432713 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-resizer-driver-longhorn-io I0227 20:14:23.117203 1 leaderelection.go:252] successfully acquired lease longhorn-system/external-resizer-driver-longhorn-io I0227 20:14:23.117241 1 leader_election.go:172] new leader detected, current leader: csi-resizer-7b5b48db77-fxflz I0227 20:14:23.117276 1 leader_election.go:165] became leader, starting I0227 20:14:23.117343 1 controller.go:190] Starting external resizer driver.longhorn.io I0227 20:14:23.117564 1 reflector.go:153] Starting reflector *v1.PersistentVolumeClaim (10m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:14:23.117564 1 reflector.go:153] Starting reflector *v1.PersistentVolume (10m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:14:23.117588 1 reflector.go:188] Listing and watching *v1.PersistentVolumeClaim from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:14:23.117593 1 reflector.go:188] Listing and watching *v1.PersistentVolume from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:14:23.217507 1 shared_informer.go:227] caches populated I0227 20:14:23.623123 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-resizer-driver-longhorn-io I0227 20:14:28.818289 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-resizer-driver-longhorn-io I0227 20:14:34.391869 1 leaderelection.go:283] successfully renewed lease longhorn-system/external-resizer-driver-longhorn-io I0227 20:14:49.392476 1 leaderelection.go:288] failed to renew lease longhorn-system/external-resizer-driver-longhorn-io: failed to tryAcquireOrRenew context deadline exceeded F0227 20:14:49.392534 1 leader_election.go:169] stopped leading

csi-snapshotter log:

I0227 20:15:53.141672 1 main.go:87] Version: v2.1.1-lh1-0-gb60fe9af0 I0227 20:15:53.146105 1 connection.go:153] Connecting to unix:///csi/csi.sock I0227 20:15:53.147476 1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo I0227 20:15:53.147493 1 connection.go:183] GRPC request: {} I0227 20:15:53.152172 1 connection.go:185] GRPC response: {“name”:“driver.longhorn.io”,“vendor_version”:“v1.1.0”} I0227 20:15:53.152715 1 connection.go:186] GRPC error: <nil> I0227 20:15:53.152722 1 main.go:136] CSI driver name: “driver.longhorn.io” W0227 20:15:53.152734 1 metrics.go:142] metrics endpoint will not be started because metrics-address was not specified. I0227 20:15:53.152746 1 common.go:111] Probing CSI driver for readiness I0227 20:15:53.152754 1 connection.go:182] GRPC call: /csi.v1.Identity/Probe I0227 20:15:53.152764 1 connection.go:183] GRPC request: {} I0227 20:15:53.155955 1 connection.go:185] GRPC response: {} I0227 20:15:53.158793 1 connection.go:186] GRPC error: <nil> I0227 20:15:53.158803 1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities I0227 20:15:53.158809 1 connection.go:183] GRPC request: {} I0227 20:15:53.167107 1 connection.go:185] GRPC response: {“capabilities”:[{“Type”:{“Rpc”:{“type”:1}}},{“Type”:{“Rpc”:{“type”:2}}},{“Type”:{“Rpc”:{“type”:9}}},{“Type”:{“Rpc”:{“type”:5}}}]} I0227 20:15:53.168871 1 connection.go:186] GRPC error: <nil> I0227 20:15:53.168880 1 main.go:162] Start NewCSISnapshotSideCarController with snapshotter [driver.longhorn.io] kubeconfig [] csiTimeout [1m0s] csiAddress [/csi/csi.sock] resyncPeriod [1m0s] snapshotNamePrefix [snapshot] snapshotNameUUIDLength [824636228488] I0227 20:15:53.169312 1 leaderelection.go:242] attempting to acquire leader lease longhorn-system/external-snapshotter-leader-driver-longhorn-io… I0227 20:15:53.288705 1 leaderelection.go:352] lock is held by csi-snapshotter-f77d6f866-8fff2 and has not yet expired I0227 20:15:53.288736 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-snapshotter-leader-driver-longhorn-io I0227 20:15:53.288752 1 leader_election.go:172] new leader detected, current leader: csi-snapshotter-f77d6f866-8fff2 I0227 20:16:02.054796 1 leaderelection.go:352] lock is held by csi-snapshotter-f77d6f866-8fff2 and has not yet expired I0227 20:16:02.054817 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-snapshotter-leader-driver-longhorn-io I0227 20:16:12.763179 1 leaderelection.go:352] lock is held by csi-snapshotter-f77d6f866-8fff2 and has not yet expired I0227 20:16:12.763221 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-snapshotter-leader-driver-longhorn-io E0227 20:16:23.292818 1 leaderelection.go:331] error retrieving resource lock longhorn-system/external-snapshotter-leader-driver-longhorn-io: etcdserver: leader changed I0227 20:16:23.292850 1 leaderelection.go:247] failed to acquire lease longhorn-system/external-snapshotter-leader-driver-longhorn-io I0227 20:16:40.281656 1 leader_election.go:172] new leader detected, current leader: csi-snapshotter-f77d6f866-56rff I0227 20:16:40.281691 1 leaderelection.go:252] successfully acquired lease longhorn-system/external-snapshotter-leader-driver-longhorn-io I0227 20:16:40.281750 1 leader_election.go:165] became leader, starting I0227 20:16:40.282015 1 reflector.go:153] Starting reflector *v1beta1.VolumeSnapshotClass (1m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:40.282043 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:40.282440 1 reflector.go:153] Starting reflector *v1beta1.VolumeSnapshotContent (1m0s) from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:40.282454 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:40.282629 1 snapshot_controller_base.go:115] Starting CSI snapshotter E0227 20:16:40.284322 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) E0227 20:16:40.284397 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) I0227 20:16:41.284480 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:41.285693 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) I0227 20:16:41.287469 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:41.289587 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) I0227 20:16:42.288372 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:42.289780 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) I0227 20:16:42.289806 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:42.290835 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) I0227 20:16:43.289943 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:43.290963 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:43.291570 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) E0227 20:16:43.293323 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) I0227 20:16:44.291783 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:44.293180 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) I0227 20:16:44.293463 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:44.294292 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) I0227 20:16:45.293395 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:45.294589 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:45.296011 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) E0227 20:16:45.296682 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) I0227 20:16:46.296263 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:46.297317 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:46.317111 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) E0227 20:16:46.317166 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) I0227 20:16:47.317319 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:47.318406 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:47.318898 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) E0227 20:16:47.319480 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) I0227 20:16:48.319091 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:48.320480 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:48.320700 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) E0227 20:16:48.321521 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) I0227 20:16:49.320880 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotClass from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 I0227 20:16:49.321997 1 reflector.go:188] Listing and watching *v1beta1.VolumeSnapshotContent from pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108 E0227 20:16:49.323308 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io) E0227 20:16:49.323308 1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1beta1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io) I0227 20:16:50.282007 1 leaderelection.go:288] failed to renew lease longhorn-system/external-snapshotter-leader-driver-longhorn-io: failed to tryAcquireOrRenew context deadline exceeded F0227 20:16:50.282046 1 leader_election.go:169] stopped leading Wrap lines Previous Container

All other workloads are OK:

Eevn I have errors on mentioned longhorn workloads in clusters that have only one node(VM) I want to know what cause this issue? I can’t deploy anything that needs persistent volume at this period .