longhorn: [BUG] VolumeSnapshot remains in `READYTOUSE=false` indefinitely.

Describe the bug I am attempting to setup Kasten K10 for backing up and snapshotting longhorn volumes to an external server. When creating a volumesnapshot, either through Kasten or manually (such as using the example provided in the docs), a volumesnapshot resource is created with an associated volumesnapshotcontent, but the resource remains in the READYTOUSE = false state indefinitely.

The Events field for all volumesnapshotcontent resources looks as follows:

Events:
  Type     Reason                               Age    From                                Message
  ----     ------                               ----   ----                                -------
  Warning  SnapshotCreationFailed               5m32s  csi-snapshotter driver.longhorn.io  Failed to create snapshot: failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
  Warning  SnapshotContentCheckandUpdateFailed  4m32s  csi-snapshotter driver.longhorn.io  Failed to check and update snapshot content: failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"

To Reproduce Steps to reproduce the behavior:

Install longhorn via helm chart
Enable CSI Snapshot support as per docs:
- Install CRDs
- Install snapshot-controller
Create VolumeSnapshotClass

apiVersion: snapshot.storage.k8s.io/v1
driver: driver.longhorn.io
kind: VolumeSnapshotClass
metadata:
  annotations:
    k10.kasten.io/is-snapshot-class: "true"
    snapshot.storage.kubernetes.io/is-default-class: "true"
  name: longhorn-snapclass
deletionPolicy: Delete

Attempt to create volumesnapshot using example manifest

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: test-snapshot-pvc
spec:
  volumeSnapshotClassName: longhorn-snapclass
  source:
    persistentVolumeClaimName: test-vol

Expected behavior Each volumesnapshot resource should transition to READYTOUSE = true after a short period of time.

Log

longhorn-support-bundle_d9405fdc-34cb-45e0-a50f-36c8b9d60679_2021-10-20T00-16-16Z.zip

Helm Values

csi:
  kubeletRootDir: /var/lib/kubelet
ingress:
  enabled: true
  ingressClassName: nginx
  host: longhorn.${DOMAIN}
  tls: true
  tlsSecret: longhorn-cert
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: ${CLUSTER_ISSUER}
defaultSettings:
  defaultDataLocality: best-effort
  replicaSoftAntiAffinity: true
  taintToleration: node-role.kubernetes.io/longhorn:NoSchedule
  replicaAutoBalance: best-effort
longhornManager: 
  tolerations:
  - key: "node-role.kubernetes.io/longhorn"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
longhornDriver:
  tolerations:
  - key: "node-role.kubernetes.io/longhorn"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"
longhornUI:
  tolerations:
  - key: "node-role.kubernetes.io/longhorn"
    operator: "Exists"
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

Snapshot Controller Logs

I1020 00:03:20.026268       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:03:25.181924       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:03:30.495647       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:03:35.563922       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:03:41.156999       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:03:42.641047       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-qsqhxx2l2p2975nm" for sync
I1020 00:03:42.641106       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:42.641123       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:42.641137       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-qsqhxx2l2p2975nm]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:42.641148       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:42.641160       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:42.641183       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:42.641195       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:42.641213       1 util.go:187] storeObjectUpdate: adding snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm", version 51499976
I1020 00:03:42.641233       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: bound to: "", Completed: false
I1020 00:03:42.641244       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:42.641259       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-qsqhxx2l2p2975nm]: snapshot status [(*v1.VolumeSnapshotStatus)(nil)]
I1020 00:03:42.641277       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:42.641314       1 snapshot_controller.go:929] Keeping PVC media/qbittorrent-config, it is used by snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:42.641329       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add invalid label on snapshot
I1020 00:03:42.641341       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: validate snapshot to make sure source has been correctly specified
I1020 00:03:42.641353       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add finalizers on snapshot
I1020 00:03:42.641365       1 snapshot_controller.go:382] checkandAddSnapshotFinalizers: Add Finalizer for VolumeSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:42.755749       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-qsqhxx2l2p2975nm" for sync
I1020 00:03:42.795800       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499983
I1020 00:03:42.795845       1 snapshot_controller.go:1412] Added protection finalizer to volume snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:42.795860       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:42.795869       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:42.795878       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:42.795900       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:42.795912       1 snapshot_controller.go:635] createSnapshotContent: Creating content for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm through the plugin ...
I1020 00:03:42.795923       1 snapshot_controller.go:638] createSnapshotContent: Check if PVC is not being deleted and add Finalizer for source of snapshot [k10-csi-snap-qsqhxx2l2p2975nm] if needed
I1020 00:03:42.795942       1 snapshot_controller.go:867] Protection finalizer already exists for persistent volume claim media/qbittorrent-config
I1020 00:03:42.795952       1 snapshot_controller.go:719] getCreateSnapshotInput [k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:42.795959       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:42.819897       1 snapshot_controller.go:1206] getVolumeFromVolumeSnapshot: snapshot [k10-csi-snap-qsqhxx2l2p2975nm] PV name [pvc-0660a1b7-1024-408f-842c-e74c47b1f362]
I1020 00:03:42.819947       1 snapshot_controller.go:684] volume snapshot content &v1.VolumeSnapshotContent{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.VolumeSnapshotContentSpec{VolumeSnapshotRef:v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"media", Name:"k10-csi-snap-qsqhxx2l2p2975nm", UID:"8ccf9e7c-2a77-4e69-9698-db5a18f86700", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"51499976", FieldPath:""}, DeletionPolicy:"Delete", Driver:"driver.longhorn.io", VolumeSnapshotClassName:(*string)(0xc0003aab60), Source:v1.VolumeSnapshotContentSource{VolumeHandle:(*string)(0xc0009149d0), SnapshotHandle:(*string)(nil)}}, Status:(*v1.VolumeSnapshotContentStatus)(nil)}
I1020 00:03:42.820040       1 snapshot_controller.go:686] createSnapshotContent [media/k10-csi-snap-qsqhxx2l2p2975nm]: trying to save volume snapshot content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700
I1020 00:03:42.858431       1 snapshot_controller_base.go:182] enqueued "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" for sync
I1020 00:03:42.858542       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]
I1020 00:03:42.858596       1 util.go:187] storeObjectUpdate: adding content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700", version 51499984
I1020 00:03:42.858620       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: content is bound to snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:42.858630       1 snapshot_controller.go:89] syncContent[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: check if we should add invalid label on content
I1020 00:03:42.858949       1 snapshot_controller.go:118] syncContent [snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: Add Finalizer for VolumeSnapshotContent
I1020 00:03:42.991383       1 snapshot_controller.go:694] volume snapshot content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" for snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" saved, &{{ } {snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {{VolumeSnapshot media k10-csi-snap-qsqhxx2l2p2975nm 8ccf9e7c-2a77-4e69-9698-db5a18f86700 snapshot.storage.k8s.io/v1 51499976 } Delete driver.longhorn.io 0xc0003aab60 {0xc0009149d0 <nil>}} <nil>}
I1020 00:03:42.991569       1 util.go:215] storeObjectUpdate updating content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" with version 51499984
I1020 00:03:42.991601       1 snapshot_controller.go:512] syncUnreadySnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: trying to update snapshot status
I1020 00:03:42.991620       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:42.991632       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-qsqhxx2l2p2975nm GenerateName: Namespace:media SelfLink: UID:8ccf9e7c-2a77-4e69-9698-db5a18f86700 ResourceVersion:51499976 Generation:1 CreationTimestamp:2021-10-20 00:03:42 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:42 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc00001c220 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc00001c230} Status:<nil>}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:42.992046       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"media", Name:"k10-csi-snap-qsqhxx2l2p2975nm", UID:"8ccf9e7c-2a77-4e69-9698-db5a18f86700", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"51499976", FieldPath:""}): type: 'Normal' reason: 'CreatingSnapshot' Waiting for a snapshot media/k10-csi-snap-qsqhxx2l2p2975nm to be created by the CSI driver.
I1020 00:03:43.073409       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.073445       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.073458       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-qsqhxx2l2p2975nm]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.073464       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.073471       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:43.073484       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:43.073500       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:43.073520       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499983
I1020 00:03:43.073548       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: bound to: "", Completed: false
I1020 00:03:43.073559       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:43.073575       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-qsqhxx2l2p2975nm]: snapshot status [(*v1.VolumeSnapshotStatus)(nil)]
I1020 00:03:43.073591       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.073630       1 snapshot_controller.go:929] Keeping PVC media/qbittorrent-config, it is used by snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.073644       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add invalid label on snapshot
I1020 00:03:43.073657       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: validate snapshot to make sure source has been correctly specified
I1020 00:03:43.073669       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add finalizers on snapshot
I1020 00:03:43.073687       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.073706       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.073714       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.073746       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.073768       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.073779       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-qsqhxx2l2p2975nm]: snapshotContent [snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]
I1020 00:03:43.106482       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: trying to update snapshot status
I1020 00:03:43.106513       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.106561       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-qsqhxx2l2p2975nm GenerateName: Namespace:media SelfLink: UID:8ccf9e7c-2a77-4e69-9698-db5a18f86700 ResourceVersion:51499987 Generation:1 CreationTimestamp:2021-10-20 00:03:42 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:42 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc000174430 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc000174440} Status:0xc00001e7b0}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:43.131356       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499987
I1020 00:03:43.131399       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-qsqhxx2l2p2975nm", GenerateName:"", Namespace:"media", SelfLink:"", UID:"8ccf9e7c-2a77-4e69-9698-db5a18f86700", ResourceVersion:"51499987", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770285022, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"qbittorrent-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc00042cee8), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00042c078), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc00042cf18), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00042cf00), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc0005f5d70), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc0005f5d80)}, Status:(*v1.VolumeSnapshotStatus)(0xc0004439e0)}]
I1020 00:03:43.131522       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-qsqhxx2l2p2975nm  media  8ccf9e7c-2a77-4e69-9698-db5a18f86700 51499987 1 2021-10-20 00:03:42 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:03:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} }]} {{0xc0005f5d70 <nil>} 0xc0005f5d80} 0xc0004439e0}
I1020 00:03:43.151714       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-qsqhxx2l2p2975nm" for sync
I1020 00:03:43.151765       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.151777       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.151789       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-qsqhxx2l2p2975nm]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.151798       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.151807       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:43.151820       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:43.151831       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:43.151849       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499987
I1020 00:03:43.151870       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: bound to: "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700", Completed: false
I1020 00:03:43.151880       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:43.151922       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-qsqhxx2l2p2975nm]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc000174a30), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc0007e939b), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(nil)}]
I1020 00:03:43.151954       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.152000       1 snapshot_controller.go:929] Keeping PVC media/qbittorrent-config, it is used by snapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:03:43.152012       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add invalid label on snapshot
I1020 00:03:43.152022       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: validate snapshot to make sure source has been correctly specified
I1020 00:03:43.152032       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add finalizers on snapshot
I1020 00:03:43.152045       1 snapshot_controller.go:382] checkandAddSnapshotFinalizers: Add Finalizer for VolumeSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.312041       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-t2vfvx8dn28v6wzl" for sync
I1020 00:03:43.312086       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.312100       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.312113       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-t2vfvx8dn28v6wzl]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.312122       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.312129       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:43.312139       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:43.312155       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:43.312171       1 util.go:187] storeObjectUpdate: adding snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl", version 51499988
I1020 00:03:43.312201       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: bound to: "", Completed: false
I1020 00:03:43.312216       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:43.312230       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-t2vfvx8dn28v6wzl]: snapshot status [(*v1.VolumeSnapshotStatus)(nil)]
I1020 00:03:43.312250       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.312357       1 snapshot_controller.go:929] Keeping PVC media/prowlarr-config, it is used by snapshot media/k10-csi-snap-95whrnn95zqfhvdf
I1020 00:03:43.312371       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add invalid label on snapshot
I1020 00:03:43.312384       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: validate snapshot to make sure source has been correctly specified
I1020 00:03:43.312396       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add finalizers on snapshot
I1020 00:03:43.312410       1 snapshot_controller.go:382] checkandAddSnapshotFinalizers: Add Finalizer for VolumeSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.320885       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499991
I1020 00:03:43.320912       1 snapshot_controller.go:1412] Added protection finalizer to volume snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.320926       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.320936       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.320942       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.320963       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.320990       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.321002       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-qsqhxx2l2p2975nm]: snapshotContent [snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]
I1020 00:03:43.360535       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51499993
I1020 00:03:43.360577       1 snapshot_controller.go:1412] Added protection finalizer to volume snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.360589       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.360598       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.360607       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.360625       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.360655       1 snapshot_controller.go:635] createSnapshotContent: Creating content for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl through the plugin ...
I1020 00:03:43.360666       1 snapshot_controller.go:638] createSnapshotContent: Check if PVC is not being deleted and add Finalizer for source of snapshot [k10-csi-snap-t2vfvx8dn28v6wzl] if needed
I1020 00:03:43.360683       1 snapshot_controller.go:867] Protection finalizer already exists for persistent volume claim media/prowlarr-config
I1020 00:03:43.360694       1 snapshot_controller.go:719] getCreateSnapshotInput [k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.360700       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.379594       1 snapshot_controller_base.go:182] enqueued "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" for sync
I1020 00:03:43.416251       1 snapshot_controller.go:1206] getVolumeFromVolumeSnapshot: snapshot [k10-csi-snap-t2vfvx8dn28v6wzl] PV name [pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af]
I1020 00:03:43.416296       1 snapshot_controller.go:684] volume snapshot content &v1.VolumeSnapshotContent{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.VolumeSnapshotContentSpec{VolumeSnapshotRef:v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"media", Name:"k10-csi-snap-t2vfvx8dn28v6wzl", UID:"36d402f8-c2f2-460d-8fe3-4f10f6dd941e", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"51499988", FieldPath:""}, DeletionPolicy:"Delete", Driver:"driver.longhorn.io", VolumeSnapshotClassName:(*string)(0xc0003aab60), Source:v1.VolumeSnapshotContentSource{VolumeHandle:(*string)(0xc0009155d0), SnapshotHandle:(*string)(nil)}}, Status:(*v1.VolumeSnapshotContentStatus)(nil)}
I1020 00:03:43.416425       1 snapshot_controller.go:686] createSnapshotContent [media/k10-csi-snap-t2vfvx8dn28v6wzl]: trying to save volume snapshot content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e
I1020 00:03:43.438222       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-qsqhxx2l2p2975nm" for sync
I1020 00:03:43.463619       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: trying to update snapshot status
I1020 00:03:43.463651       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.463692       1 snapshot_controller_base.go:182] enqueued "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" for sync
I1020 00:03:43.463738       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]
I1020 00:03:43.463757       1 util.go:187] storeObjectUpdate: adding content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e", version 51499995
I1020 00:03:43.463790       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: content is bound to snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.463664       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-qsqhxx2l2p2975nm GenerateName: Namespace:media SelfLink: UID:8ccf9e7c-2a77-4e69-9698-db5a18f86700 ResourceVersion:51499991 Generation:1 CreationTimestamp:2021-10-20 00:03:42 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:42 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc000175460 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc000175470} Status:0xc00001f4d0}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:43.463809       1 snapshot_controller.go:89] syncContent[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: check if we should add invalid label on content
I1020 00:03:43.463819       1 snapshot_controller.go:118] syncContent [snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: Add Finalizer for VolumeSnapshotContent
I1020 00:03:43.466379       1 snapshot_controller.go:694] volume snapshot content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" for snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" saved, &{{ } {snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] []  []} {{VolumeSnapshot media k10-csi-snap-t2vfvx8dn28v6wzl 36d402f8-c2f2-460d-8fe3-4f10f6dd941e snapshot.storage.k8s.io/v1 51499988 } Delete driver.longhorn.io 0xc0003aab60 {0xc0009155d0 <nil>}} <nil>}
I1020 00:03:43.466503       1 util.go:215] storeObjectUpdate updating content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" with version 51499995
I1020 00:03:43.466530       1 snapshot_controller.go:512] syncUnreadySnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: trying to update snapshot status
I1020 00:03:43.466543       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.466585       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshot", Namespace:"media", Name:"k10-csi-snap-t2vfvx8dn28v6wzl", UID:"36d402f8-c2f2-460d-8fe3-4f10f6dd941e", APIVersion:"snapshot.storage.k8s.io/v1", ResourceVersion:"51499988", FieldPath:""}): type: 'Normal' reason: 'CreatingSnapshot' Waiting for a snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl to be created by the CSI driver.
I1020 00:03:43.466554       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-t2vfvx8dn28v6wzl GenerateName: Namespace:media SelfLink: UID:36d402f8-c2f2-460d-8fe3-4f10f6dd941e ResourceVersion:51499988 Generation:1 CreationTimestamp:2021-10-20 00:03:43 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:prowlarr-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc0006184b0 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc0006184c0} Status:<nil>}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:43.502110       1 snapshot_controller_base.go:182] enqueued "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" for sync
I1020 00:03:43.502630       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-t2vfvx8dn28v6wzl" for sync
I1020 00:03:43.503406       1 util.go:215] storeObjectUpdate updating content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" with version 51499997
I1020 00:03:43.503429       1 snapshot_controller.go:825] Added protection finalizer to volume snapshot content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e
I1020 00:03:43.503470       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]
I1020 00:03:43.503481       1 util.go:215] storeObjectUpdate updating content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" with version 51499997
I1020 00:03:43.503493       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: content is bound to snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.503502       1 snapshot_controller.go:89] syncContent[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: check if we should add invalid label on content
I1020 00:03:43.503510       1 snapshot_controller.go:1480] getSnapshotFromStore: snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl found
I1020 00:03:43.503521       1 snapshot_controller.go:1042] needsUpdateSnapshotStatus[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.586140       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499991
I1020 00:03:43.586206       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-qsqhxx2l2p2975nm", GenerateName:"", Namespace:"media", SelfLink:"", UID:"8ccf9e7c-2a77-4e69-9698-db5a18f86700", ResourceVersion:"51499991", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770285022, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"qbittorrent-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumesnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc0007cc180), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0007cc168), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc0007cc1b0), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0007cc198), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc00001c1b0), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc00001c1c0)}, Status:(*v1.VolumeSnapshotStatus)(0xc0007461e0)}]
I1020 00:03:43.586617       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-qsqhxx2l2p2975nm  media  8ccf9e7c-2a77-4e69-9698-db5a18f86700 51499991 1 2021-10-20 00:03:42 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:03:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} }]} {{0xc00001c1b0 <nil>} 0xc00001c1c0} 0xc0007461e0}
I1020 00:03:43.586948       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.586962       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.586975       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-qsqhxx2l2p2975nm]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.586983       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.586994       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:43.587008       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:43.587020       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-qsqhxx2l2p2975nm"
I1020 00:03:43.587033       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499991
I1020 00:03:43.587054       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: bound to: "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700", Completed: false
I1020 00:03:43.587065       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:43.587236       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-qsqhxx2l2p2975nm]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc0001752a0), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc0007e9c7b), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(nil)}]
I1020 00:03:43.587265       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.587297       1 snapshot_controller.go:929] Keeping PVC media/qbittorrent-config, it is used by snapshot media/k10-csi-snap-z4qb4wq8p9kkvvrl
I1020 00:03:43.587308       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add invalid label on snapshot
I1020 00:03:43.587318       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: validate snapshot to make sure source has been correctly specified
I1020 00:03:43.587330       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-qsqhxx2l2p2975nm]: check if we should add finalizers on snapshot
I1020 00:03:43.587346       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.587353       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.587359       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.587379       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.587787       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 for snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.587811       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-qsqhxx2l2p2975nm]: snapshotContent [snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]
I1020 00:03:43.641647       1 request.go:597] Waited for 138.632541ms due to client-side throttling, not priority and fairness, request: PUT:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl/status
I1020 00:03:43.697837       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.697871       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.697888       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-t2vfvx8dn28v6wzl]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.697895       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.697902       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:43.697911       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:43.697922       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:43.697941       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51499993
I1020 00:03:43.697961       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: bound to: "", Completed: false
I1020 00:03:43.697970       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:43.697981       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-t2vfvx8dn28v6wzl]: snapshot status [(*v1.VolumeSnapshotStatus)(nil)]
I1020 00:03:43.697996       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.698023       1 snapshot_controller.go:929] Keeping PVC media/prowlarr-config, it is used by snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.698041       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add invalid label on snapshot
I1020 00:03:43.698053       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: validate snapshot to make sure source has been correctly specified
I1020 00:03:43.698066       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add finalizers on snapshot
I1020 00:03:43.698087       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.698130       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:43.698142       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:43.698156       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.698168       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:43.698181       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-t2vfvx8dn28v6wzl]: snapshotContent [snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]
I1020 00:03:43.801592       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-t2vfvx8dn28v6wzl" for sync
I1020 00:03:43.841693       1 request.go:597] Waited for 253.76605ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:43.886931       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-qsqhxx2l2p2975nm]: trying to update snapshot status
I1020 00:03:43.886961       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:43.886970       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-qsqhxx2l2p2975nm GenerateName: Namespace:media SelfLink: UID:8ccf9e7c-2a77-4e69-9698-db5a18f86700 ResourceVersion:51499991 Generation:1 CreationTimestamp:2021-10-20 00:03:42 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:42 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc000174790 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc0001747a0} Status:0xc00001e990}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:44.041940       1 request.go:597] Waited for 343.605906ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.061783       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: trying to update snapshot status
I1020 00:03:44.061822       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:44.061834       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-t2vfvx8dn28v6wzl GenerateName: Namespace:media SelfLink: UID:36d402f8-c2f2-460d-8fe3-4f10f6dd941e ResourceVersion:51500003 Generation:1 CreationTimestamp:2021-10-20 00:03:43 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:prowlarr-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc000618cf0 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc000618d00} Status:0xc00018b740}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:44.130121       1 util.go:215] storeObjectUpdate updating content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" with version 51499994
I1020 00:03:44.130156       1 snapshot_controller.go:825] Added protection finalizer to volume snapshot content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700
I1020 00:03:44.130194       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]
I1020 00:03:44.130203       1 util.go:215] storeObjectUpdate updating content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" with version 51499994
I1020 00:03:44.130213       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: content is bound to snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:44.130219       1 snapshot_controller.go:89] syncContent[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: check if we should add invalid label on content
I1020 00:03:44.130229       1 snapshot_controller.go:1480] getSnapshotFromStore: snapshot media/k10-csi-snap-qsqhxx2l2p2975nm found
I1020 00:03:44.130245       1 snapshot_controller.go:1042] needsUpdateSnapshotStatus[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:03:44.242544       1 request.go:597] Waited for 355.376176ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:03:44.282992       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-qsqhxx2l2p2975nm" with version 51499991
I1020 00:03:44.283035       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-qsqhxx2l2p2975nm", GenerateName:"", Namespace:"media", SelfLink:"", UID:"8ccf9e7c-2a77-4e69-9698-db5a18f86700", ResourceVersion:"51499991", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770285022, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"qbittorrent-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumesnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc00000d278), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d260), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc00000d2f0), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d2c0), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc000174f20), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc000174f30)}, Status:(*v1.VolumeSnapshotStatus)(0xc00001efc0)}]
I1020 00:03:44.283209       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-qsqhxx2l2p2975nm  media  8ccf9e7c-2a77-4e69-9698-db5a18f86700 51499991 1 2021-10-20 00:03:42 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:03:42 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} }]} {{0xc000174f20 <nil>} 0xc000174f30} 0xc00001efc0}
I1020 00:03:44.441666       1 request.go:597] Waited for 379.532362ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.458366       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51500003
I1020 00:03:44.458395       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-t2vfvx8dn28v6wzl", GenerateName:"", Namespace:"media", SelfLink:"", UID:"36d402f8-c2f2-460d-8fe3-4f10f6dd941e", ResourceVersion:"51500003", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770285023, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"prowlarr-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc00000d530), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d518), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc00000d560), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00000d548), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc000175600), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc000175610)}, Status:(*v1.VolumeSnapshotStatus)(0xc00001f2f0)}]
I1020 00:03:44.458501       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-t2vfvx8dn28v6wzl  media  36d402f8-c2f2-460d-8fe3-4f10f6dd941e 51500003 1 2021-10-20 00:03:43 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:prowlarr-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} }]} {{0xc000175600 <nil>} 0xc000175610} 0xc00001f2f0}
I1020 00:03:44.458622       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:44.458639       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:44.458649       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-t2vfvx8dn28v6wzl]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:44.458656       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:44.458662       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:44.458672       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:44.458686       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:44.458700       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51500003
I1020 00:03:44.458733       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: bound to: "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e", Completed: false
I1020 00:03:44.458747       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:44.458761       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-t2vfvx8dn28v6wzl]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc0006189e0), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc00075341b), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(nil)}]
I1020 00:03:44.458782       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:44.458809       1 snapshot_controller.go:929] Keeping PVC media/prowlarr-config, it is used by snapshot media/k10-csi-snap-pplqt59wffsd6mqq
I1020 00:03:44.458818       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add invalid label on snapshot
I1020 00:03:44.458826       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: validate snapshot to make sure source has been correctly specified
I1020 00:03:44.458833       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add finalizers on snapshot
I1020 00:03:44.458848       1 snapshot_controller.go:382] checkandAddSnapshotFinalizers: Add Finalizer for VolumeSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:44.642302       1 request.go:597] Waited for 183.208079ms due to client-side throttling, not priority and fairness, request: PUT:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.678606       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-t2vfvx8dn28v6wzl" for sync
I1020 00:03:44.680033       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51500014
I1020 00:03:44.680070       1 snapshot_controller.go:1412] Added protection finalizer to volume snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.680086       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.680096       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:44.680105       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:44.680128       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.680227       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.680283       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-t2vfvx8dn28v6wzl]: snapshotContent [snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]
I1020 00:03:44.841651       1 request.go:597] Waited for 161.23835ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:44.871519       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: trying to update snapshot status
I1020 00:03:44.871561       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:44.871574       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-t2vfvx8dn28v6wzl GenerateName: Namespace:media SelfLink: UID:36d402f8-c2f2-460d-8fe3-4f10f6dd941e ResourceVersion:51500014 Generation:1 CreationTimestamp:2021-10-20 00:03:43 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:prowlarr-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc0000c8250 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc0000c8260} Status:0xc000018f60}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:45.042046       1 request.go:597] Waited for 170.272501ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:45.061535       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51500014
I1020 00:03:45.061567       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-t2vfvx8dn28v6wzl", GenerateName:"", Namespace:"media", SelfLink:"", UID:"36d402f8-c2f2-460d-8fe3-4f10f6dd941e", ResourceVersion:"51500014", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770285023, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"prowlarr-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumesnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc00042db30), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00042db18), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc00042db60), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00042db48), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc000619560), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc000619570)}, Status:(*v1.VolumeSnapshotStatus)(0xc00018be60)}]
I1020 00:03:45.061697       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-t2vfvx8dn28v6wzl  media  36d402f8-c2f2-460d-8fe3-4f10f6dd941e 51500014 1 2021-10-20 00:03:43 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:prowlarr-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} }]} {{0xc000619560 <nil>} 0xc000619570} 0xc00018be60}
I1020 00:03:45.061817       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:45.061832       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:45.061845       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-t2vfvx8dn28v6wzl]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:45.061854       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:45.061861       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:03:45.061873       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:45.061886       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl"
I1020 00:03:45.061895       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51500014
I1020 00:03:45.061913       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: bound to: "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e", Completed: false
I1020 00:03:45.061928       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:03:45.061939       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-t2vfvx8dn28v6wzl]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc0005a08f0), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc0005aa39b), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(nil)}]
I1020 00:03:45.061963       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:45.061992       1 snapshot_controller.go:929] Keeping PVC media/prowlarr-config, it is used by snapshot media/k10-csi-snap-8ws6s9zd7kstzd5x
I1020 00:03:45.061999       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add invalid label on snapshot
I1020 00:03:45.062007       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: validate snapshot to make sure source has been correctly specified
I1020 00:03:45.062014       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-t2vfvx8dn28v6wzl]: check if we should add finalizers on snapshot
I1020 00:03:45.062027       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:45.062035       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:45.062044       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:03:45.062062       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:45.062078       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e for snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:45.062084       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-t2vfvx8dn28v6wzl]: snapshotContent [snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]
I1020 00:03:45.242445       1 request.go:597] Waited for 180.264857ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:45.263318       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-t2vfvx8dn28v6wzl]: trying to update snapshot status
I1020 00:03:45.263341       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:03:45.263349       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-t2vfvx8dn28v6wzl GenerateName: Namespace:media SelfLink: UID:36d402f8-c2f2-460d-8fe3-4f10f6dd941e ResourceVersion:51500014 Generation:1 CreationTimestamp:2021-10-20 00:03:43 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:prowlarr-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:43 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc0005a0af0 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc0005a0b00} Status:0xc000616180}] based on VolumeSnapshotContentStatus [<nil>]
I1020 00:03:45.441840       1 request.go:597] Waited for 178.284628ms due to client-side throttling, not priority and fairness, request: GET:https://10.43.0.1:443/apis/snapshot.storage.k8s.io/v1/namespaces/media/volumesnapshots/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:03:45.463850       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-t2vfvx8dn28v6wzl" with version 51500014
I1020 00:03:45.463884       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-t2vfvx8dn28v6wzl", GenerateName:"", Namespace:"media", SelfLink:"", UID:"36d402f8-c2f2-460d-8fe3-4f10f6dd941e", ResourceVersion:"51500014", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770285023, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"prowlarr-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumesnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc0001a2150), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0001a2138), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc0001a2228), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0001a2168), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc000256370), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc000256380)}, Status:(*v1.VolumeSnapshotStatus)(0xc000513290)}]
I1020 00:03:45.463995       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-t2vfvx8dn28v6wzl  media  36d402f8-c2f2-460d-8fe3-4f10f6dd941e 51500014 1 2021-10-20 00:03:43 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:prowlarr-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:43 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:readyToUse":{}}} }]} {{0xc000256370 <nil>} 0xc000256380} 0xc000513290}
I1020 00:03:46.218523       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:03:51.281895       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:03:56.391540       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:01.461094       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:06.122154       1 snapshot_controller_base.go:182] enqueued "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" for sync
I1020 00:04:06.122199       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]
I1020 00:04:06.122219       1 util.go:215] storeObjectUpdate updating content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700" with version 51500191
I1020 00:04:06.122237       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: content is bound to snapshot media/k10-csi-snap-qsqhxx2l2p2975nm
I1020 00:04:06.122256       1 snapshot_controller.go:89] syncContent[snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: check if we should add invalid label on content
I1020 00:04:06.122267       1 snapshot_controller.go:1480] getSnapshotFromStore: snapshot media/k10-csi-snap-qsqhxx2l2p2975nm found
I1020 00:04:06.122276       1 snapshot_controller.go:1042] needsUpdateSnapshotStatus[media/k10-csi-snap-qsqhxx2l2p2975nm]
I1020 00:04:06.571482       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:11.624618       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:16.698693       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:19.006097       1 snapshot_controller_base.go:182] enqueued "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" for sync
I1020 00:04:19.006139       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]
I1020 00:04:19.006157       1 util.go:215] storeObjectUpdate updating content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e" with version 51500289
I1020 00:04:19.006173       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: content is bound to snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl
I1020 00:04:19.006183       1 snapshot_controller.go:89] syncContent[snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: check if we should add invalid label on content
I1020 00:04:19.006195       1 snapshot_controller.go:1480] getSnapshotFromStore: snapshot media/k10-csi-snap-t2vfvx8dn28v6wzl found
I1020 00:04:19.006207       1 snapshot_controller.go:1042] needsUpdateSnapshotStatus[media/k10-csi-snap-t2vfvx8dn28v6wzl]
I1020 00:04:21.783635       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:26.828936       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:31.895930       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:33.301913       1 reflector.go:535] github.com/kubernetes-csi/external-snapshotter/client/v4/informers/externalversions/factory.go:117: Watch close - *v1.VolumeSnapshot total 41 items received
I1020 00:04:35.600123       1 snapshot_controller_base.go:182] enqueued "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a" for sync
I1020 00:04:35.600161       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]
I1020 00:04:35.600178       1 util.go:215] storeObjectUpdate updating content "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a" with version 51500435
I1020 00:04:35.600196       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]: content is bound to snapshot media/k10-csi-snap-95whrnn95zqfhvdf
I1020 00:04:35.600206       1 snapshot_controller.go:89] syncContent[snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]: check if we should add invalid label on content
I1020 00:04:35.600220       1 snapshot_controller.go:1480] getSnapshotFromStore: snapshot media/k10-csi-snap-95whrnn95zqfhvdf found
I1020 00:04:35.600234       1 snapshot_controller.go:1042] needsUpdateSnapshotStatus[media/k10-csi-snap-95whrnn95zqfhvdf]
I1020 00:04:35.600245       1 snapshot_controller.go:142] synchronizing VolumeSnapshotContent for snapshot [media/k10-csi-snap-95whrnn95zqfhvdf]: update snapshot status to true if needed.
I1020 00:04:35.600303       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-95whrnn95zqfhvdf]
I1020 00:04:35.600317       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-95whrnn95zqfhvdf]
I1020 00:04:35.600328       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-95whrnn95zqfhvdf]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:35.600339       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:35.600349       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:04:35.600367       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-95whrnn95zqfhvdf"
I1020 00:04:35.600379       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-95whrnn95zqfhvdf"
I1020 00:04:35.600393       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-95whrnn95zqfhvdf" with version 51499719
I1020 00:04:35.600409       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-95whrnn95zqfhvdf]: bound to: "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a", Completed: false
I1020 00:04:35.600416       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-95whrnn95zqfhvdf]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:04:35.600443       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-95whrnn95zqfhvdf]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc000a12bb0), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc00053d7a8), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(0xc000a12bc0)}]
I1020 00:04:35.600471       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-95whrnn95zqfhvdf]
I1020 00:04:35.600502       1 snapshot_controller.go:929] Keeping PVC media/prowlarr-config, it is used by snapshot media/k10-csi-snap-mk6jgzzn28ktvbt7
I1020 00:04:35.600514       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-95whrnn95zqfhvdf]: check if we should add invalid label on snapshot
I1020 00:04:35.600523       1 snapshot_controller.go:238] processSnapshotWithDeletionTimestamp VolumeSnapshot[media/k10-csi-snap-95whrnn95zqfhvdf]: bound to: "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a", Completed: false
I1020 00:04:35.600537       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-95whrnn95zqfhvdf]
I1020 00:04:35.600542       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:35.600553       1 snapshot_controller.go:286] processSnapshotWithDeletionTimestamp[media/k10-csi-snap-95whrnn95zqfhvdf]: delete snapshot content and remove finalizer from snapshot if needed
I1020 00:04:35.600563       1 snapshot_controller.go:293] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent VolumeSnapshot[media/k10-csi-snap-95whrnn95zqfhvdf]: bound to: "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a", Completed: false
I1020 00:04:35.600581       1 snapshot_controller.go:847] isVolumeBeingCreatedFromSnapshot: no volume is being created from snapshot media/k10-csi-snap-95whrnn95zqfhvdf
I1020 00:04:35.600593       1 snapshot_controller.go:312] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent[media/k10-csi-snap-95whrnn95zqfhvdf]: Set VolumeSnapshotBeingDeleted annotation on the content [snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]
I1020 00:04:35.600603       1 snapshot_controller.go:325] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: set DeletionTimeStamp on content [snapcontent-f74efddf-04ca-4644-815a-08855ec7455a].
I1020 00:04:35.856078       1 snapshot_controller.go:333] checkandRemoveSnapshotFinalizersAndCheckandDeleteContent: Remove Finalizer for VolumeSnapshot[media/k10-csi-snap-95whrnn95zqfhvdf]
I1020 00:04:35.856102       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-95whrnn95zqfhvdf]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc000a12bb0), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc00053d7a8), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(0xc000a12bc0)}]
I1020 00:04:35.856136       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-95whrnn95zqfhvdf]
I1020 00:04:35.856162       1 snapshot_controller.go:929] Keeping PVC media/prowlarr-config, it is used by snapshot media/k10-csi-snap-rwxhzfvmmjppcxfz
I1020 00:04:35.891927       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-95whrnn95zqfhvdf" with version 51499719
I1020 00:04:35.891961       1 snapshot_controller.go:1456] Removed protection finalizer from volume snapshot media/k10-csi-snap-95whrnn95zqfhvdf
I1020 00:04:36.173694       1 snapshot_controller_base.go:182] enqueued "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445" for sync
I1020 00:04:36.173741       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]
I1020 00:04:36.173760       1 util.go:215] storeObjectUpdate updating content "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445" with version 51500438
I1020 00:04:36.173780       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]: content is bound to snapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:04:36.173798       1 snapshot_controller.go:89] syncContent[snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]: check if we should add invalid label on content
I1020 00:04:36.173810       1 snapshot_controller.go:1480] getSnapshotFromStore: snapshot media/k10-csi-snap-kf9m45q2q56cxhsr found
I1020 00:04:36.173827       1 snapshot_controller.go:1042] needsUpdateSnapshotStatus[media/k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.173837       1 snapshot_controller.go:142] synchronizing VolumeSnapshotContent for snapshot [media/k10-csi-snap-kf9m45q2q56cxhsr]: update snapshot status to true if needed.
I1020 00:04:36.173883       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.173895       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.173904       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-kf9m45q2q56cxhsr]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:36.173911       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:36.173921       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:04:36.173939       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-kf9m45q2q56cxhsr"
I1020 00:04:36.173949       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-kf9m45q2q56cxhsr"
I1020 00:04:36.173958       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-kf9m45q2q56cxhsr" with version 51499790
I1020 00:04:36.173976       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: bound to: "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445", Completed: false
I1020 00:04:36.173991       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-kf9m45q2q56cxhsr]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:04:36.174004       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-kf9m45q2q56cxhsr]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc0002e0520), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc000839478), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(0xc0002e0540)}]
I1020 00:04:36.174040       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.174076       1 snapshot_controller.go:929] Keeping PVC media/qbittorrent-config, it is used by snapshot media/k10-csi-snap-qf59s57pz744xlgv
I1020 00:04:36.174095       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: check if we should add invalid label on snapshot
I1020 00:04:36.174107       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: validate snapshot to make sure source has been correctly specified
I1020 00:04:36.174119       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: check if we should add finalizers on snapshot
I1020 00:04:36.174135       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:04:36.174145       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.174152       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:36.174171       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:04:36.174191       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 for snapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:04:36.174206       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-kf9m45q2q56cxhsr]: snapshotContent [snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]
I1020 00:04:36.213413       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-kf9m45q2q56cxhsr]: trying to update snapshot status
I1020 00:04:36.213441       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.213453       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-kf9m45q2q56cxhsr GenerateName: Namespace:media SelfLink: UID:1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 ResourceVersion:51499790 Generation:1 CreationTimestamp:2021-10-20 00:00:33 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:00:33 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:19 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc0004ae850 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc0004ae860} Status:0xc000617f20}] based on VolumeSnapshotContentStatus [&{SnapshotHandle:<nil> CreationTime:<nil> RestoreSize:<nil> ReadyToUse:0xc0004d3b40 Error:0xc000a21520}]
I1020 00:04:36.274326       1 snapshot_controller_base.go:165] enqueued "media/k10-csi-snap-kf9m45q2q56cxhsr" for sync
I1020 00:04:36.671578       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-kf9m45q2q56cxhsr" with version 51500440
I1020 00:04:36.671610       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-kf9m45q2q56cxhsr", GenerateName:"", Namespace:"media", SelfLink:"", UID:"1ef4a8e1-1bdc-4c7e-b461-4de99fa14445", ResourceVersion:"51500440", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770284833, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"qbittorrent-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumesnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc0008a2d08), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0008a2cf0), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc0008a2d38), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0008a2d20), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc00038e860), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc00038e880)}, Status:(*v1.VolumeSnapshotStatus)(0xc00016f1d0)}]
I1020 00:04:36.671714       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-kf9m45q2q56cxhsr  media  1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 51500440 1 2021-10-20 00:00:33 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:00:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:19 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUse":{}}} }]} {{0xc00038e860 <nil>} 0xc00038e880} 0xc00016f1d0}
I1020 00:04:36.671848       1 snapshot_controller_base.go:209] syncSnapshotByKey[media/k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.671865       1 snapshot_controller_base.go:212] snapshotWorker: snapshot namespace [media] name [k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.671878       1 snapshot_controller_base.go:335] checkAndUpdateSnapshotClass [k10-csi-snap-kf9m45q2q56cxhsr]: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:36.671892       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:36.671903       1 snapshot_controller_base.go:355] VolumeSnapshotClass [longhorn-snapclass] Driver [driver.longhorn.io]
I1020 00:04:36.671917       1 snapshot_controller_base.go:229] Updating snapshot "media/k10-csi-snap-kf9m45q2q56cxhsr"
I1020 00:04:36.671932       1 snapshot_controller_base.go:365] updateSnapshot "media/k10-csi-snap-kf9m45q2q56cxhsr"
I1020 00:04:36.671950       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-kf9m45q2q56cxhsr" with version 51500440
I1020 00:04:36.671972       1 snapshot_controller.go:182] synchronizing VolumeSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: bound to: "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445", Completed: false
I1020 00:04:36.671989       1 snapshot_controller.go:184] syncSnapshot [media/k10-csi-snap-kf9m45q2q56cxhsr]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I1020 00:04:36.672001       1 snapshot_controller.go:954] checkandRemovePVCFinalizer for snapshot [k10-csi-snap-kf9m45q2q56cxhsr]: snapshot status [&v1.VolumeSnapshotStatus{BoundVolumeSnapshotContentName:(*string)(0xc0004af160), CreationTime:(*v1.Time)(nil), ReadyToUse:(*bool)(0xc0009167f8), RestoreSize:(*resource.Quantity)(nil), Error:(*v1.VolumeSnapshotError)(0xc0004af170)}]
I1020 00:04:36.672027       1 snapshot_controller.go:909] Checking isPVCBeingUsed for snapshot [media/k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.672079       1 snapshot_controller.go:929] Keeping PVC media/qbittorrent-config, it is used by snapshot media/k10-csi-snap-z4qb4wq8p9kkvvrl
I1020 00:04:36.672095       1 snapshot_controller.go:193] syncSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: check if we should add invalid label on snapshot
I1020 00:04:36.672108       1 snapshot_controller.go:209] syncSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: validate snapshot to make sure source has been correctly specified
I1020 00:04:36.672123       1 snapshot_controller.go:218] syncSnapshot[media/k10-csi-snap-kf9m45q2q56cxhsr]: check if we should add finalizers on snapshot
I1020 00:04:36.672146       1 snapshot_controller.go:417] syncUnreadySnapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:04:36.672188       1 snapshot_controller.go:1259] getSnapshotDriverName: VolumeSnapshot[k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.672195       1 snapshot_controller.go:1245] getSnapshotClass: VolumeSnapshotClassName [longhorn-snapclass]
I1020 00:04:36.672211       1 snapshot_controller.go:478] getDynamicallyProvisionedContentFromStore for snapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:04:36.672229       1 snapshot_controller.go:486] Found VolumeSnapshotContent object snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 for snapshot media/k10-csi-snap-kf9m45q2q56cxhsr
I1020 00:04:36.672242       1 snapshot_controller.go:1008] bindandUpdateVolumeSnapshot for snapshot [k10-csi-snap-kf9m45q2q56cxhsr]: snapshotContent [snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]
I1020 00:04:36.757291       1 snapshot_controller.go:1018] bindandUpdateVolumeSnapshot [media/k10-csi-snap-kf9m45q2q56cxhsr]: trying to update snapshot status
I1020 00:04:36.757318       1 snapshot_controller.go:1074] updateSnapshotStatus[media/k10-csi-snap-kf9m45q2q56cxhsr]
I1020 00:04:36.757332       1 snapshot_controller.go:1095] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k10-csi-snap-kf9m45q2q56cxhsr GenerateName: Namespace:media SelfLink: UID:1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 ResourceVersion:51500440 Generation:1 CreationTimestamp:2021-10-20 00:00:33 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] Annotations:map[] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection] ClusterName: ManagedFields:[{Manager:executor-server Operation:Update APIVersion:snapshot.storage.k8s.io/v1beta1 Time:2021-10-20 00:00:33 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2021-10-20 00:03:19 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUse":{}}} Subresource:}]} Spec:{Source:{PersistentVolumeClaimName:0xc000257de0 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc000257df0} Status:0xc000163b30}] based on VolumeSnapshotContentStatus [&{SnapshotHandle:<nil> CreationTime:<nil> RestoreSize:<nil> ReadyToUse:0xc0004d3b40 Error:0xc000a21520}]
I1020 00:04:36.892037       1 util.go:215] storeObjectUpdate updating snapshot "media/k10-csi-snap-kf9m45q2q56cxhsr" with version 51500440
I1020 00:04:36.892067       1 snapshot_controller.go:1035] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"k10-csi-snap-kf9m45q2q56cxhsr", GenerateName:"", Namespace:"media", SelfLink:"", UID:"1ef4a8e1-1bdc-4c7e-b461-4de99fa14445", ResourceVersion:"51500440", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63770284833, loc:(*time.Location)(0x26410c0)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"kasten_io_appnamespace":"media", "kasten_io_jobid":"47567034-3138-11ec-95b9-668b37020b9a", "kasten_io_manifestid":"474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5", "kasten_io_pvc":"qbittorrent-config", "name":"kasten__snapshot-media-ns-2021-10-20t00-00-00z-00"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumesnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"executor-server", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1beta1", Time:(*v1.Time)(0xc0007e2030), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0007e2018), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:(*v1.Time)(0xc0007e2090), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0007e2048), Subresource:""}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClaimName:(*string)(0xc000a12400), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc000a12410)}, Status:(*v1.VolumeSnapshotStatus)(0xc0005f21e0)}]
I1020 00:04:36.892166       1 snapshot_controller.go:496] bindandUpdateVolumeSnapshot &{{ } {k10-csi-snap-kf9m45q2q56cxhsr  media  1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 51500440 1 2021-10-20 00:00:33 +0000 UTC <nil> <nil> map[kasten_io_appnamespace:media kasten_io_jobid:47567034-3138-11ec-95b9-668b37020b9a kasten_io_manifestid:474ee5d2-3138-11ec-b5e2-2ec5a8e45cb5 kasten_io_pvc:qbittorrent-config name:kasten__snapshot-media-ns-2021-10-20t00-00-00z-00] map[] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{executor-server Update snapshot.storage.k8s.io/v1beta1 2021-10-20 00:00:33 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:kasten_io_appnamespace":{},"f:kasten_io_jobid":{},"f:kasten_io_manifestid":{},"f:kasten_io_pvc":{},"f:name":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update snapshot.storage.k8s.io/v1 2021-10-20 00:03:19 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{}},"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUse":{}}} }]} {{0xc000a12400 <nil>} 0xc000a12410} 0xc0005f21e0}
I1020 00:04:36.987526       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I1020 00:04:40.264911       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolumeClaim total 12 items received
I1020 00:04:42.095803       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader

CSI Snapshotter Logs

I1020 00:03:53.602927       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c through the plugin ...
I1020 00:03:55.658662       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:03:55.658722       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-88d046ab-1810-4180-a65b-60d06ab50331]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:03:55.658741       1 snapshot_controller_base.go:261] could not sync content "snapcontent-88d046ab-1810-4180-a65b-60d06ab50331": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:03:55.658805       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc through the plugin ...
I1020 00:03:57.452891       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:03:57.452947       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:03:57.452966       1 snapshot_controller_base.go:261] could not sync content "snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:03:57.453037       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a86d66f9-f76a-4337-8731-d29703822852 through the plugin ...
I1020 00:03:57.547440       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:03:57.547495       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:03:57.547513       1 snapshot_controller_base.go:261] could not sync content "snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:03:57.547581       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b through the plugin ...
I1020 00:04:06.060050       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:06.060135       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:06.060155       1 snapshot_controller_base.go:261] could not sync content "snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:06.060217       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 through the plugin ...
W1020 00:04:06.121671       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
I1020 00:04:18.938247       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:18.938310       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3]: error occurred failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:18.938326       1 snapshot_controller_base.go:261] could not sync content "snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3": failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:18.938381       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e through the plugin ...
W1020 00:04:19.008600       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
I1020 00:04:19.750980       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-654bd10b-8482-4a70-a772-c78524bbada2 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:19.751039       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-654bd10b-8482-4a70-a772-c78524bbada2]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:19.751058       1 snapshot_controller_base.go:261] could not sync content "snapcontent-654bd10b-8482-4a70-a772-c78524bbada2": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:19.751147       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-54765ee3-331a-4356-8579-aee01859eef2 through the plugin ...
I1020 00:04:29.078168       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:29.078218       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:29.078237       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:29.078289       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 through the plugin ...
I1020 00:04:35.551089       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f74efddf-04ca-4644-815a-08855ec7455a returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:04:35.601869       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:04:35.602062       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:35.602083       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:35.602124       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 through the plugin ...
I1020 00:04:35.602160       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-f74efddf-04ca-4644-815a-08855ec7455a", UID:"4c263b69-0fc0-4966-b8d7-082de9b78ddc", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"51500435", FieldPath:""}): type: 'Warning' reason: 'SnapshotContentCheckandUpdateFailed' Failed to check and update snapshot content: failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:36.128161       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:04:36.173989       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:04:36.174154       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:36.174179       1 snapshot_controller_base.go:261] could not sync content "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:36.174223       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 through the plugin ...
I1020 00:04:36.174274       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445", UID:"2bbd8d76-788b-48d5-8733-c9060e28f67e", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"51500438", FieldPath:""}): type: 'Warning' reason: 'SnapshotContentCheckandUpdateFailed' Failed to check and update snapshot content: failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
W1020 00:04:38.682038       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
I1020 00:04:53.603175       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:53.603444       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:53.603502       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:53.603610       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5 through the plugin ...
I1020 00:04:55.659083       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:55.659186       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:55.659210       1 snapshot_controller_base.go:261] could not sync content "snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:55.659281       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3 through the plugin ...
I1020 00:04:57.453377       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-a86d66f9-f76a-4337-8731-d29703822852 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:57.453438       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-a86d66f9-f76a-4337-8731-d29703822852]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:57.453454       1 snapshot_controller_base.go:261] could not sync content "snapcontent-a86d66f9-f76a-4337-8731-d29703822852": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:57.453529       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-654bd10b-8482-4a70-a772-c78524bbada2 through the plugin ...
I1020 00:04:57.548505       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:04:57.549495       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:04:57.549530       1 snapshot_controller_base.go:261] could not sync content "snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:04:57.549643       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485 through the plugin ...
I1020 00:05:06.122069       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:05:06.164918       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:05:06.165079       1 snapshot_controller.go:105] createSnapshot for content [snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: error occurred in createSnapshotWrapper: failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:06.165111       1 snapshot_controller_base.go:261] could not sync content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:06.165173       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f74efddf-04ca-4644-815a-08855ec7455a through the plugin ...
I1020 00:05:06.165307       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"", UID:"", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SnapshotCreationFailed' Failed to create snapshot: failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
W1020 00:05:10.744257       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotClass is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotClass
I1020 00:05:19.009014       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:05:19.043173       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:05:19.043312       1 snapshot_controller.go:105] createSnapshot for content [snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: error occurred in createSnapshotWrapper: failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:19.043334       1 snapshot_controller_base.go:261] could not sync content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:19.043399       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 through the plugin ...
I1020 00:05:19.043434       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"", UID:"", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'SnapshotCreationFailed' Failed to create snapshot: failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:19.751467       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-54765ee3-331a-4356-8579-aee01859eef2 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:19.751692       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-54765ee3-331a-4356-8579-aee01859eef2]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:19.751718       1 snapshot_controller_base.go:261] could not sync content "snapcontent-54765ee3-331a-4356-8579-aee01859eef2": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:19.751816       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c through the plugin ...
I1020 00:05:29.078536       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:29.078584       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-88d046ab-1810-4180-a65b-60d06ab50331]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:29.078605       1 snapshot_controller_base.go:261] could not sync content "snapcontent-88d046ab-1810-4180-a65b-60d06ab50331": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:29.078685       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc through the plugin ...
I1020 00:05:35.602380       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:35.602447       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:35.602464       1 snapshot_controller_base.go:261] could not sync content "snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:35.602535       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a86d66f9-f76a-4337-8731-d29703822852 through the plugin ...
I1020 00:05:36.174461       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:36.174521       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:36.174540       1 snapshot_controller_base.go:261] could not sync content "snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:36.174620       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b through the plugin ...
I1020 00:05:53.603884       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:53.604003       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:53.604028       1 snapshot_controller_base.go:261] could not sync content "snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:53.604110       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 through the plugin ...
I1020 00:05:55.659494       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:55.659559       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3]: error occurred failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:55.659576       1 snapshot_controller_base.go:261] could not sync content "snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3": failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:55.659640       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e through the plugin ...
I1020 00:05:57.453776       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-654bd10b-8482-4a70-a772-c78524bbada2 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:57.453838       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-654bd10b-8482-4a70-a772-c78524bbada2]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:57.453860       1 snapshot_controller_base.go:261] could not sync content "snapcontent-654bd10b-8482-4a70-a772-c78524bbada2": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:57.453926       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-54765ee3-331a-4356-8579-aee01859eef2 through the plugin ...
I1020 00:05:57.550149       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:05:57.550233       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:05:57.550283       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:05:57.550444       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 through the plugin ...
I1020 00:06:06.165404       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f74efddf-04ca-4644-815a-08855ec7455a returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:06.165454       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:06.165475       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:06.165544       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 through the plugin ...
I1020 00:06:19.043633       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:19.043702       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:19.043723       1 snapshot_controller_base.go:261] could not sync content "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:19.043787       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 through the plugin ...
I1020 00:06:19.752043       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:19.752104       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:19.752126       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:19.752196       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5 through the plugin ...
I1020 00:06:29.079159       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:29.079383       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:29.079450       1 snapshot_controller_base.go:261] could not sync content "snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:29.079577       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3 through the plugin ...
I1020 00:06:35.602800       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-a86d66f9-f76a-4337-8731-d29703822852 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:35.602856       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-a86d66f9-f76a-4337-8731-d29703822852]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:35.602876       1 snapshot_controller_base.go:261] could not sync content "snapcontent-a86d66f9-f76a-4337-8731-d29703822852": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:35.602944       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-654bd10b-8482-4a70-a772-c78524bbada2 through the plugin ...
I1020 00:06:36.174840       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:36.174988       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:36.175011       1 snapshot_controller_base.go:261] could not sync content "snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:36.175071       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485 through the plugin ...
I1020 00:06:53.604343       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:06:53.653406       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:06:53.653612       1 snapshot_controller.go:105] createSnapshot for content [snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: error occurred in createSnapshotWrapper: failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:53.653636       1 snapshot_controller_base.go:261] could not sync content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:53.653675       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f74efddf-04ca-4644-815a-08855ec7455a through the plugin ...
I1020 00:06:53.653669       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700", UID:"3d9dee31-8461-44b5-9954-83250843bf0a", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"51501583", FieldPath:""}): type: 'Warning' reason: 'SnapshotCreationFailed' Failed to create snapshot: failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:55.659925       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:06:55.718989       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:06:55.719364       1 snapshot_controller.go:105] createSnapshot for content [snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: error occurred in createSnapshotWrapper: failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:55.719462       1 snapshot_controller_base.go:261] could not sync content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:55.719556       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 through the plugin ...
I1020 00:06:55.720629       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e", UID:"482f20ff-bf50-4db8-9c3e-01a87bf406dc", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"51501611", FieldPath:""}): type: 'Warning' reason: 'SnapshotCreationFailed' Failed to create snapshot: failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:57.454200       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-54765ee3-331a-4356-8579-aee01859eef2 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:57.454296       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-54765ee3-331a-4356-8579-aee01859eef2]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:57.454358       1 snapshot_controller_base.go:261] could not sync content "snapcontent-54765ee3-331a-4356-8579-aee01859eef2": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:57.454431       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c through the plugin ...
I1020 00:06:57.550785       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:06:57.551047       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-88d046ab-1810-4180-a65b-60d06ab50331]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:06:57.551111       1 snapshot_controller_base.go:261] could not sync content "snapcontent-88d046ab-1810-4180-a65b-60d06ab50331": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:06:57.551268       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc through the plugin ...
I1020 00:07:06.165787       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:06.165867       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:06.165889       1 snapshot_controller_base.go:261] could not sync content "snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:06.165957       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a86d66f9-f76a-4337-8731-d29703822852 through the plugin ...
I1020 00:07:19.044006       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:19.044076       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:19.044093       1 snapshot_controller_base.go:261] could not sync content "snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:19.044156       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b through the plugin ...
I1020 00:07:19.752426       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:19.752564       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:19.752606       1 snapshot_controller_base.go:261] could not sync content "snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:19.752666       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 through the plugin ...
I1020 00:07:29.079800       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:29.079863       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3]: error occurred failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:29.079882       1 snapshot_controller_base.go:261] could not sync content "snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3": failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:29.079951       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e through the plugin ...
I1020 00:07:35.603195       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-654bd10b-8482-4a70-a772-c78524bbada2 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:35.603248       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-654bd10b-8482-4a70-a772-c78524bbada2]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:35.603267       1 snapshot_controller_base.go:261] could not sync content "snapcontent-654bd10b-8482-4a70-a772-c78524bbada2": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:35.603328       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-54765ee3-331a-4356-8579-aee01859eef2 through the plugin ...
I1020 00:07:36.175338       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:36.175397       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:36.175417       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:36.175517       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 through the plugin ...
I1020 00:07:53.653917       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f74efddf-04ca-4644-815a-08855ec7455a returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:53.653966       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:53.653982       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:53.654042       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 through the plugin ...
I1020 00:07:55.719842       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:55.719904       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:55.719923       1 snapshot_controller_base.go:261] could not sync content "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:55.720000       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 through the plugin ...
I1020 00:07:57.454673       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:57.454732       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:57.454749       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:57.454806       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5 through the plugin ...
I1020 00:07:57.552256       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:07:57.552433       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:07:57.552455       1 snapshot_controller_base.go:261] could not sync content "snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:07:57.552553       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3 through the plugin ...
I1020 00:08:06.166205       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-a86d66f9-f76a-4337-8731-d29703822852 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:06.166267       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-a86d66f9-f76a-4337-8731-d29703822852]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:06.166289       1 snapshot_controller_base.go:261] could not sync content "snapcontent-a86d66f9-f76a-4337-8731-d29703822852": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:06.166365       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-654bd10b-8482-4a70-a772-c78524bbada2 through the plugin ...
I1020 00:08:19.044344       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:19.044403       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:19.044423       1 snapshot_controller_base.go:261] could not sync content "snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:19.044489       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485 through the plugin ...
I1020 00:08:19.752906       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:08:19.799269       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:08:19.799445       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:19.799467       1 snapshot_controller_base.go:261] could not sync content "snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:19.799509       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f74efddf-04ca-4644-815a-08855ec7455a through the plugin ...
I1020 00:08:19.799547       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700", UID:"3d9dee31-8461-44b5-9954-83250843bf0a", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"51502314", FieldPath:""}): type: 'Warning' reason: 'SnapshotContentCheckandUpdateFailed' Failed to check and update snapshot content: failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:29.080189       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
W1020 00:08:29.128195       1 warnings.go:67] snapshot.storage.k8s.io/v1beta1 VolumeSnapshotContent is deprecated; use snapshot.storage.k8s.io/v1 VolumeSnapshotContent
E1020 00:08:29.128395       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:29.128416       1 snapshot_controller_base.go:261] could not sync content "snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:29.128456       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 through the plugin ...
I1020 00:08:29.128578       1 event.go:282] Event(v1.ObjectReference{Kind:"VolumeSnapshotContent", Namespace:"", Name:"snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e", UID:"482f20ff-bf50-4db8-9c3e-01a87bf406dc", APIVersion:"snapshot.storage.k8s.io/v1beta1", ResourceVersion:"51502391", FieldPath:""}): type: 'Warning' reason: 'SnapshotContentCheckandUpdateFailed' Failed to check and update snapshot content: failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:35.603564       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-54765ee3-331a-4356-8579-aee01859eef2 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:35.603626       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-54765ee3-331a-4356-8579-aee01859eef2]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:35.603644       1 snapshot_controller_base.go:261] could not sync content "snapcontent-54765ee3-331a-4356-8579-aee01859eef2": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:35.603705       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-f597b2ef-9383-4a3e-b83d-838c224a907c through the plugin ...
I1020 00:08:36.175774       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:36.175825       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-88d046ab-1810-4180-a65b-60d06ab50331]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:36.175845       1 snapshot_controller_base.go:261] could not sync content "snapcontent-88d046ab-1810-4180-a65b-60d06ab50331": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:36.175898       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-229774ba-6084-4e11-8ec0-0d38774818dc through the plugin ...
I1020 00:08:53.655074       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:53.655421       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:53.655448       1 snapshot_controller_base.go:261] could not sync content "snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:53.655536       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a86d66f9-f76a-4337-8731-d29703822852 through the plugin ...
I1020 00:08:55.720205       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:55.720267       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:55.720294       1 snapshot_controller_base.go:261] could not sync content "snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:55.720358       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-a6fd43f9-db32-4e4e-8c92-6f140f01680b through the plugin ...
I1020 00:08:57.455009       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:57.455075       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:57.455093       1 snapshot_controller_base.go:261] could not sync content "snapcontent-b8660753-f871-4aea-92ba-eaa865a96cd5": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:57.455172       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-8ccf9e7c-2a77-4e69-9698-db5a18f86700 through the plugin ...
I1020 00:08:57.554075       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:08:57.554446       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3]: error occurred failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:08:57.554506       1 snapshot_controller_base.go:261] could not sync content "snapcontent-e865e29b-4481-4ae7-a507-f2726b3efdf3": failed to take snapshot of the volume, pvc-446f532d-e055-4baa-80fd-cd55db94cbe0: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:08:57.554586       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-36d402f8-c2f2-460d-8fe3-4f10f6dd941e through the plugin ...
I1020 00:09:06.166586       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-654bd10b-8482-4a70-a772-c78524bbada2 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:09:06.166655       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-654bd10b-8482-4a70-a772-c78524bbada2]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:09:06.166675       1 snapshot_controller_base.go:261] could not sync content "snapcontent-654bd10b-8482-4a70-a772-c78524bbada2": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:09:06.166738       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-54765ee3-331a-4356-8579-aee01859eef2 through the plugin ...
I1020 00:09:19.044683       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:09:19.044744       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:09:19.044761       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f64a0375-c20b-41aa-a883-2c96c6777485": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:09:19.044930       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-88d046ab-1810-4180-a65b-60d06ab50331 through the plugin ...
I1020 00:09:19.799737       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-f74efddf-04ca-4644-815a-08855ec7455a returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:09:19.799791       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-f74efddf-04ca-4644-815a-08855ec7455a]: error occurred failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:09:19.799808       1 snapshot_controller_base.go:261] could not sync content "snapcontent-f74efddf-04ca-4644-815a-08855ec7455a": failed to take snapshot of the volume, pvc-78cac62e-be0e-42d4-9eb0-9709eaac95af: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:09:19.799866       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-65efe734-9281-4aae-910c-98c9fa8c7fa4 through the plugin ...
I1020 00:09:29.128695       1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445 returned error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E1020 00:09:29.128757       1 snapshot_controller.go:122] checkandUpdateContentStatus [snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445]: error occurred failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
E1020 00:09:29.128772       1 snapshot_controller_base.go:261] could not sync content "snapcontent-1ef4a8e1-1bdc-4c7e-b461-4de99fa14445": failed to take snapshot of the volume, pvc-0660a1b7-1024-408f-842c-e74c47b1f362: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
I1020 00:09:29.128826       1 snapshot_controller.go:267] createSnapshotWrapper: Creating snapshot for content snapcontent-803cb37a-b5cb-4c26-916d-a77961b48b93 through the plugin ...

Environment: My cluster is comprised on 5 nodes, 4x RPi 4 4GB + 1x Dell R610. The Pis serve only as control plane nodes and don’t run any workloads or storage. All workloads and longhorn volumes run on the R610.

Longhorn version: 1.2.2
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: K3s - v1.20.7+k3s1
- Number of management node in the cluster: 3
- Number of worker node in the cluster: 2
Node config (R610)
- OS type and version: Debian 10
- CPU per node: 24 CPU
- Memory per node: 80GB
- Disk type(e.g. SSD/NVMe): SATA SSD
- Network bandwidth between the nodes: 1Gbe
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Baremetal
Number of Longhorn volumes in the cluster: 42

Thanks.

About this issue

Original URL
State: open
Created 3 years ago
Reactions: 1
Comments: 17 (8 by maintainers)

Most upvoted comments

You need to enable backups for it to work, so either just install minio or set up a NFS share. Test the backup and delete all the snapshots you have. Then it should start to work.

modzilla99 on Jan 6, 2022

Hello everyone, I’m currently experiencing the same issue. Environment: rancher running on k3s and child clusters running on RKE 1.21.7 running with longhorn to provision PVCs.

I followed the documentation on deploying longhorn with helm:

## Namespace longhorn-system
NAME                                        READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATES
csi-attacher-5f46994f7-5nkpk                1/1     Running   0          43m   10.42.205.76   sdb-opsw-003   <none>           <none>
csi-attacher-5f46994f7-h28p2                1/1     Running   0          43m   10.42.12.10    sdb-opsw-002   <none>           <none>
csi-attacher-5f46994f7-jrhmz                1/1     Running   0          43m   10.42.205.75   sdb-opsw-003   <none>           <none>
csi-provisioner-6ccbfbf86f-grth6            1/1     Running   0          43m   10.42.12.12    sdb-opsw-002   <none>           <none>
csi-provisioner-6ccbfbf86f-qsxpz            1/1     Running   0          43m   10.42.12.11    sdb-opsw-002   <none>           <none>
csi-provisioner-6ccbfbf86f-vmnx2            1/1     Running   0          43m   10.42.205.77   sdb-opsw-003   <none>           <none>
csi-resizer-6dd8bd4c97-9f4bm                1/1     Running   0          43m   10.42.205.78   sdb-opsw-003   <none>           <none>
csi-resizer-6dd8bd4c97-fbk8x                1/1     Running   0          43m   10.42.12.13    sdb-opsw-002   <none>           <none>
csi-resizer-6dd8bd4c97-hzn57                1/1     Running   0          43m   10.42.12.14    sdb-opsw-002   <none>           <none>
csi-snapshotter-86f65d8bc-l4l2f             1/1     Running   0          43m   10.42.12.15    sdb-opsw-002   <none>           <none>
csi-snapshotter-86f65d8bc-mm5d5             1/1     Running   0          43m   10.42.205.79   sdb-opsw-003   <none>           <none>
csi-snapshotter-86f65d8bc-s56gr             1/1     Running   0          43m   10.42.205.81   sdb-opsw-003   <none>           <none>
engine-image-ei-fa2dfbf0-8f8jv              1/1     Running   0          43m   10.42.205.73   sdb-opsw-003   <none>           <none>
engine-image-ei-fa2dfbf0-tnmbn              1/1     Running   0          43m   10.42.12.6     sdb-opsw-002   <none>           <none>
instance-manager-e-71d3bb99                 1/1     Running   0          43m   10.42.12.7     sdb-opsw-002   <none>           <none>
instance-manager-e-d5e7be34                 1/1     Running   0          43m   10.42.205.72   sdb-opsw-003   <none>           <none>
instance-manager-r-9618de7e                 1/1     Running   0          43m   10.42.12.8     sdb-opsw-002   <none>           <none>
instance-manager-r-ee17357c                 1/1     Running   0          43m   10.42.205.74   sdb-opsw-003   <none>           <none>
longhorn-csi-plugin-2mpr9                   2/2     Running   0          43m   10.42.12.16    sdb-opsw-002   <none>           <none>
longhorn-csi-plugin-vwd88                   2/2     Running   0          43m   10.42.205.80   sdb-opsw-003   <none>           <none>
longhorn-driver-deployer-6db849975f-s4dxs   1/1     Running   0          43m   10.42.12.5     sdb-opsw-002   <none>           <none>
longhorn-manager-mqp89                      1/1     Running   0          43m   10.42.205.71   sdb-opsw-003   <none>           <none>
longhorn-manager-zs8g4                      1/1     Running   1          43m   10.42.12.3     sdb-opsw-002   <none>           <none>
longhorn-ui-6f547c964-dmt7h                 1/1     Running   0          43m   10.42.12.4     sdb-opsw-002   <none>           <none>

I also deployed csi snapshotter and snapshot controller in the default namespace:

NAME                                   READY   STATUS    RESTARTS   AGE
csi-snapshotter-0                      3/3     Running   0          42m
snapshot-controller-6d67b9b5c4-r769d   1/1     Running   0          42m
snapshot-controller-6d67b9b5c4-r8lcm   1/1     Running   0          42m

Logs from csi-snapshotter looks fine:

I0106 14:28:42.495468       1 snapshot_controller_base.go:144] enqueued "snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf" for sync
I0106 14:28:42.495491       1 snapshot_controller_base.go:178] syncContentByKey[snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf]

And the logs from snapshot-controller

I0106 14:29:10.354965       1 snapshot_controller.go:89] syncContent[snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf]: check if we should add invalid label on content
I0106 14:29:10.354969       1 snapshot_controller.go:1537] getSnapshotFromStore: snapshot default/test-snapshot5 found
I0106 14:29:10.354973       1 snapshot_controller.go:1068] needsUpdateSnapshotStatus[default/test-snapshot5]
I0106 14:29:10.756246       1 snapshot_controller_base.go:182] enqueued "snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf" for sync
I0106 14:29:10.756264       1 snapshot_controller_base.go:284] syncContentByKey[snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf]
I0106 14:29:10.756272       1 util.go:215] storeObjectUpdate updating content "snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf" with version 25223
I0106 14:29:10.756278       1 snapshot_controller.go:87] synchronizing VolumeSnapshotContent[snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf]: content is bound to snapshot default/test-snapshot5

Still the volumesnapshot is waiting for something…

Normal CreatingSnapshot 32m snapshot-controller Waiting for a snapshot default/test-snapshot5 to be created by the CSI driver.

And I can see no events at all in the snapshotcontent…

Although, I could find an error in longhorn’s csi-snapshotter I0106 14:33:24.957017 1 snapshot_controller.go:291] createSnapshotWrapper: CreateSnapshot for content snapcontent-b4df0ad1-b865-4f5c-a0b9-f7b23da074bf returned error: rpc error: code = Internal desc = Action [snapshotList] not available on [&{pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0 volume map[self:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0] map[activate:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=activate attach:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=attach cancelExpansion:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=cancelExpansion detach:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=detach engineUpgrade:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=engineUpgrade expand:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=expand pvCreate:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=pvCreate pvcCreate:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=pvcCreate recurringJobAdd:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=recurringJobAdd recurringJobDelete:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=recurringJobDelete recurringJobList:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=recurringJobList replicaRemove:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=replicaRemove updateAccessMode:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=updateAccessMode updateDataLocality:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=updateDataLocality updateReplicaAutoBalance:http://longhorn-backend:9500/v1/volumes/pvc-8ca068fc-724e-4c4c-babc-c03702fa80b0?action=updateReplicaAutoBalance]}]

Has anyone succeded in resolving this problem?

Thanking you all in advance,

sthartmannsopra on Jan 6, 2022

Could you please help dump the BackupTarget and Backup CRs?

kubectl get backuptargets.longhorn.io -n longhorn-system -o yaml
kubectl get backups.longhorn.io -n longhorn-system -o yaml

Besides that, I saw the log flooded with engine image is not ready, could you please checks the following nodes are Ready?

hyperion.saturn.kube.markmckessock.com
phoebe.saturn.kube.markmckessock.com
titan.saturn.kube.markmckessock.com

And the engine-image-ei-xxx Pods are in Running state? Because I saw the log

Engine image is not fully deployed on all nodes: 2 of 5

jenting on Oct 26, 2021