external-snapshotter: csi-snapshotter doesn't make delete request to the the CSI driver controller

What happened: We have a CSI driver that puts the restriction of creating only 10 snapshots for a volume. For the 11th snapshot this error is thrown:
“Cannot create a snapshot for volume 10972e2c-2120-4f73-914c-16fa1b906b27 because it would exceed the maximum limit of 10 snapshots”

The scenario that is giving the issue is,

1. Create 11 snapshots back to back. 10 snapshots get created and 1 remains in pending state.
2. Delete all the 11 snapshots previously created.
3. Wait for all the snapshots to be deleted.
4. Create 11 more snapshots.

When 11 more snapshots are created at step 4, the csi driver only allows 9 snapshots to be created. It happens because, when we delete 11 snapshots in step 2, 1 snapshot that remained in pending earlier, gets created. But, since we have made a delete call for all 11 the snapshots, ideally there should also have been 11 delete calls coming to the csi driver. But, there were only 10. This happened because a delete was processed by the csi-snapshotter without ever making a delete call to the csi driver controller.

The logs for that snapshot content (snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4) from the csi-snapshotter are as below:

E0620 20:58:00.925305 1 snapshot_controller.go:124] checkandUpdateContentStatus [snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4]: error occurred failed to take snapshot of the volume 10972e2c-2120-4f73-914c-16fa1b906b27: “rpc error: code = Internal desc = Operation failed: GenericOperation("error in response: status code ‘503 Service Unavailable’, content: ‘RestJsonError { details: \"Cannot create a snapshot for volume 10972e2c-2120-4f73-914c-16fa1b906b27 because it would exceed the maximum limit of 10 snapshots\", message: \"SvcError :: MaxNumberOfSnapshotsReached\", kind: Aborted }’")” E0620 20:58:00.925319 1 snapshot_controller_base.go:283] could not sync content “snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”: failed to take snapshot of the volume 10972e2c-2120-4f73-914c-16fa1b906b27: “rpc error: code = Internal desc = Operation failed: GenericOperation("error in response: status code ‘503 Service Unavailable’, content: ‘RestJsonError { details: \"Cannot create a snapshot for volume 10972e2c-2120-4f73-914c-16fa1b906b27 because it would exceed the maximum limit of 10 snapshots\", message: \"SvcError :: MaxNumberOfSnapshotsReached\", kind: Aborted }’")” W0620 20:58:02.384196 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:02.393936 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:03.346840 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:03.363092 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:04.536854 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:04.546510 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:04.551201 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:04.551278 1 snapshot_controller_base.go:283] could not sync content “snapcontent-a555ed87-fbbe-473f-b062-759b4da75c87”: snapshot controller failed to update snapcontent-a555ed87-fbbe-473f-b062-759b4da75c87 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-a555ed87-fbbe-473f-b062-759b4da75c87”: StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshotcontents/snapcontent-a555ed87-fbbe-473f-b062-759b4da75c87, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: fa4fa4db-66e9-4d83-9079-0a030e8f7882, UID in object meta: W0620 20:58:04.938250 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:04.947835 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:05.536418 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:05.546458 1 warnings.go:70] unknown field “spec.sourceVolumeMode” I0620 20:58:05.551408 1 snapshot_controller_base.go:231] deletion of content “snapcontent-a555ed87-fbbe-473f-b062-759b4da75c87” was already processed W0620 20:58:05.560769 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:05.560852 1 snapshot_controller_base.go:283] could not sync content “snapcontent-87488381-d108-45c0-af8e-9be9c4792b9c”: snapshot controller failed to update snapcontent-87488381-d108-45c0-af8e-9be9c4792b9c on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-87488381-d108-45c0-af8e-9be9c4792b9c”: StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshotcontents/snapcontent-87488381-d108-45c0-af8e-9be9c4792b9c, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c830a752-c7ac-41e5-b4a0-188bbc79aad9, UID in object meta: W0620 20:58:05.738206 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:05.753660 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:05.758330 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:05.758428 1 snapshot_controller_base.go:283] could not sync content “snapcontent-3c2d0574-9d3c-4e9a-8577-53edba766030”: snapshot controller failed to update snapcontent-3c2d0574-9d3c-4e9a-8577-53edba766030 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-3c2d0574-9d3c-4e9a-8577-53edba766030”: StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshotcontents/snapcontent-3c2d0574-9d3c-4e9a-8577-53edba766030, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 0761af43-1103-4924-aa28-9742da7b1cf7, UID in object meta: I0620 20:58:05.913160 1 snapshot_controller.go:291] createSnapshotWrapper: Creating snapshot for content snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4 through the plugin … W0620 20:58:06.144987 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:06.334466 1 warnings.go:70] unknown field “spec.sourceVolumeMode” I0620 20:58:06.561722 1 snapshot_controller_base.go:231] deletion of content “snapcontent-87488381-d108-45c0-af8e-9be9c4792b9c” was already processed W0620 20:58:06.735916 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:06.736014 1 snapshot_controller_base.go:283] could not sync content “snapcontent-952e2121-1c1d-4a3c-a9c5-432fbfad963c”: snapshot controller failed to update snapcontent-952e2121-1c1d-4a3c-a9c5-432fbfad963c on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-952e2121-1c1d-4a3c-a9c5-432fbfad963c”: StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshotcontents/snapcontent-952e2121-1c1d-4a3c-a9c5-432fbfad963c, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: dcba6998-1903-481d-bcbe-d385a121b66c, UID in object meta: I0620 20:58:06.759253 1 snapshot_controller_base.go:231] deletion of content “snapcontent-3c2d0574-9d3c-4e9a-8577-53edba766030” was already processed W0620 20:58:06.932251 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:07.334073 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:07.732433 1 warnings.go:70] unknown field “spec.sourceVolumeMode” I0620 20:58:07.736571 1 snapshot_controller_base.go:231] deletion of content “snapcontent-952e2121-1c1d-4a3c-a9c5-432fbfad963c” was already processed W0620 20:58:07.932603 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:08.135284 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:08.336134 1 warnings.go:70] unknown field “spec.sourceVolumeMode” W0620 20:58:08.530300 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:08.530410 1 snapshot_controller_base.go:283] could not sync content “snapcontent-069e931d-ec7a-4ff6-b821-6f8abbf1a77c”: snapshot controller failed to update snapcontent-069e931d-ec7a-4ff6-b821-6f8abbf1a77c on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-069e931d-ec7a-4ff6-b821-6f8abbf1a77c”: StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshotcontents/snapcontent-069e931d-ec7a-4ff6-b821-6f8abbf1a77c, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 57fd3de7-67d2-4fa2-8dc7-572681b2bb2b, UID in object meta: W0620 20:58:08.729312 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:08.729411 1 snapshot_controller_base.go:283] could not sync content “snapcontent-ee5c0a54-5964-42b7-8131-681045ec5bde”: snapshot controller failed to update snapcontent-ee5c0a54-5964-42b7-8131-681045ec5bde on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-ee5c0a54-5964-42b7-8131-681045ec5bde”: StorageError: invalid object, Code: 4, Key: /registry/snapshot.storage.k8s.io/volumesnapshotcontents/snapcontent-ee5c0a54-5964-42b7-8131-681045ec5bde, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: b4e7ce2d-5749-4c97-b629-645130377330, UID in object meta: I0620 20:58:09.531439 1 snapshot_controller_base.go:231] deletion of content “snapcontent-069e931d-ec7a-4ff6-b821-6f8abbf1a77c” was already processed I0620 20:58:09.729814 1 snapshot_controller_base.go:231] deletion of content “snapcontent-ee5c0a54-5964-42b7-8131-681045ec5bde” was already processed I0620 20:58:10.924598 1 snapshot_controller.go:325] createSnapshotWrapper: CreateSnapshot for content snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4 returned error: rpc error: code = Internal desc = Operation failed: ServerCommunication(“error in request: request timed out”) W0620 20:58:10.932729 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:10.946174 1 snapshot_controller.go:124] checkandUpdateContentStatus [snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4]: error occurred failed to remove VolumeSnapshotBeingCreated annotation from the content snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4: snapshot controller failed to update snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”: the object has been modified; please apply your changes to the latest version and try again E0620 20:58:10.946190 1 snapshot_controller_base.go:283] could not sync content “snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”: failed to remove VolumeSnapshotBeingCreated annotation from the content snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4: snapshot controller failed to update snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”: the object has been modified; please apply your changes to the latest version and try again I0620 20:58:10.946279 1 event.go:285] Event(v1.ObjectReference{Kind:“VolumeSnapshotContent”, Namespace:“”, Name:“snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”, UID:“d49b3dda-f6ad-4ccc-9023-867245f99f63”, APIVersion:“snapshot.storage.k8s.io/v1”, ResourceVersion:“4323472”, FieldPath:“”}): type: ‘Warning’ reason: ‘SnapshotContentCheckandUpdateFailed’ Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation from the content snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4: snapshot controller failed to update snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io “snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”: the object has been modified; please apply your changes to the latest version and try again I0620 20:58:10.949738 1 snapshot_controller.go:291] createSnapshotWrapper: Creating snapshot for content snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4 through the plugin … I0620 20:58:12.211550 1 snapshot_controller.go:325] createSnapshotWrapper: CreateSnapshot for content snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4 returned error: rpc error: code = Internal desc = Operation failed: ResourceAlreadyExists(“error in response: status code ‘422 Unprocessable Entity’, content: ‘RestJsonError { details: "id: 10972e2c-2120-4f73-914c-16fa1b906b27@snapshot-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4", message: "SvcError :: AlreadyExists", kind: AlreadyExists }’”) W0620 20:58:12.219334 1 warnings.go:70] unknown field “spec.sourceVolumeMode” E0620 20:58:12.227497 1 snapshot_controller.go:124] checkandUpdateContentStatus [snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4]: error occurred failed to take snapshot of the volume 10972e2c-2120-4f73-914c-16fa1b906b27: “rpc error: code = Internal desc = Operation failed: ResourceAlreadyExists("error in response: status code ‘422 Unprocessable Entity’, content: ‘RestJsonError { details: \"id: 10972e2c-2120-4f73-914c-16fa1b906b27@snapshot-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4\", message: \"SvcError :: AlreadyExists\", kind: AlreadyExists }’")” E0620 20:58:12.227515 1 snapshot_controller_base.go:283] could not sync content “snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”: failed to take snapshot of the volume 10972e2c-2120-4f73-914c-16fa1b906b27: “rpc error: code = Internal desc = Operation failed: ResourceAlreadyExists("error in response: status code ‘422 Unprocessable Entity’, content: ‘RestJsonError { details: \"id: 10972e2c-2120-4f73-914c-16fa1b906b27@snapshot-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4\", message: \"SvcError :: AlreadyExists\", kind: AlreadyExists }’")” I0620 20:58:12.227588 1 event.go:285] Event(v1.ObjectReference{Kind:“VolumeSnapshotContent”, Namespace:“”, Name:“snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4”, UID:“d49b3dda-f6ad-4ccc-9023-867245f99f63”, APIVersion:“snapshot.storage.k8s.io/v1”, ResourceVersion:“4323481”, FieldPath:“”}): type: ‘Warning’ reason: ‘SnapshotContentCheckandUpdateFailed’ Failed to check and update snapshot content: failed to take snapshot of the volume 10972e2c-2120-4f73-914c-16fa1b906b27: “rpc error: code = Internal desc = Operation failed: ResourceAlreadyExists("error in response: status code ‘422 Unprocessable Entity’, content: ‘RestJsonError { details: \"id: 10972e2c-2120-4f73-914c-16fa1b906b27@snapshot-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4\", message: \"SvcError :: AlreadyExists\", kind: AlreadyExists }’")” W0620 20:58:12.238151 1 warnings.go:70] unknown field “spec.sourceVolumeMode” I0620 20:58:13.228289 1 snapshot_controller_base.go:231] deletion of content “snapcontent-f1e1b7ea-7797-4829-a8bd-32fb34d4cee4” was already processed

What you expected to happen:

All the delete snap content calls should be made from the csi-snapshotter to the csi driver controller to maintain consistency.

How to reproduce it:

Anything else we need to know?:

Environment:

• Driver version:
• Kubernetes version (use kubectl version):
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others: