external-snapshotter: Retry interval of failed volume snapshot creation or deletion does not double after each failure: v6.0.1

What happened:

Created VolumeSnapshot but the request failed due to authorisation issue in storage provider. In logs, I could see frequent calls to CreateSnapshot method. Ad per doc, it should double after each failure but logs doesn’t show that behaviour.

What you expected to happen: retry interval of failed volume snapshot creation or deletion should double after each failure

How to reproduce it:

Create a negative test scenario where VolumeSnapshot Creation fails and observe csi-snapshotter sidecar logs.

Anything else we need to know?:

cat csi-snapshotter.log | grep "GRPC call: /csi.v1.Controller/CreateSnapshot"
I1103 09:23:10.412875       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:12.306660       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:12.477653       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:12.785050       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:12.943098       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:13.124844       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:13.700102       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:14.300774       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:14.904655       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:15.504961       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:16.101649       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:16.704480       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:17.303818       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:17.904957       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:18.524977       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:19.100899       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:19.701028       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:20.302357       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:20.903176       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:21.502685       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:22.101407       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:22.701141       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:23.306374       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:23.902031       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:24.546378       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:25.104353       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:25.706340       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:26.304882       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:26.902224       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:27.500703       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:28.100920       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:28.705816       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot

csi-snapshotter has errors logged

I1103 09:23:10.412875       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateSnapshot
I1103 09:23:10.412881       1 connection.go:184] GRPC request: {"name":"snapshot-b9eb23d0-cd0d-47e9-a73f-da6d2132bde1","source_volume_id":"r014-41301bc5-7acf-4013-aaae-386ae5cbd5f4"}
I1103 09:23:10.426415       1 snapshot_controller_base.go:162] enqueued "snapcontent-b9eb23d0-cd0d-47e9-a73f-da6d2132bde1" for sync
I1103 09:23:12.279782       1 connection.go:186] GRPC response: {}
I1103 09:23:12.279876       1 connection.go:187] GRPC error: rpc error: code = Internal desc = {RequestID: 556ec084-4b68-4bd7-bd9f-fcc5f44bccaf , Code: InternalError, Description: Internal error occurred%!(EXTRA string=creation), BackendError: {Code:SnapshotSpaceOrderFailed, Type:ProvisioningFailed, Description:Snapshot space order failed for the given volume ID, BackendError:Trace Code:556ec084-4b68-4bd7-bd9f-fcc5f44bccaf, The user is not authorized Please check Check access permissions and try again., RC:500}, Action: Please check 'BackendError' tag for more details}
I1103 09:23:12.279895       1 snapshot_controller.go:324] createSnapshotWrapper: CreateSnapshot for content snapcontent-b9eb23d0-cd0d-47e9-a73f-da6d2132bde1 returned error: rpc error: code = Internal desc = {RequestID: 556ec084-4b68-4bd7-bd9f-fcc5f44bccaf , Code: InternalError, Description: Internal error occurred%!(EXTRA string=creation), BackendError: {Code:SnapshotSpaceOrderFailed, Type:ProvisioningFailed, Description:Snapshot space order failed for the given volume ID, BackendError:Trace Code:556ec084-4b68-4bd7-bd9f-fcc5f44bccaf, The user is not authorized Please check Check access permissions and try again., RC:500}, Action: Please check 'BackendError' tag for more details}

External-Snapshotter version : v6.0.1

Related PR for reference : https://github.com/kubernetes-csi/external-snapshotter/pull/651

Environment:

• Driver version:
• Kubernetes version (use kubectl version): 1.25
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others: