velero: Can't delete snapshot after `velero backup delete`

What steps did you take and what happened: I have a backup of my PVs, and I can see these snapshots on my aws console. When I execute this command to delete this backupvelero backup delete <my-backup>. The backup is deleted, the metadata on S3 is deleted, but the snapshot still exists on my aws. I can see the logs of this process:

time="2021-11-19T07:07:30Z" level=info msg="Removing existing deletion requests for backup" backup=velero-backup-pvs-20211119063547 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:455" name=velero-backup-pvs-20211119063547-dhxjw namespace=default
time="2021-11-19T07:07:31Z" level=info msg="Removing PV snapshots" backup=velero-backup-pvs-20211119063547 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:331" name=velero-backup-pvs-20211119063547-dhxjw namespace=default
time="2021-11-19T07:07:33Z" level=info msg="Removing restic snapshots" backup=velero-backup-pvs-20211119063547 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:357" name=velero-backup-pvs-20211119063547-dhxjw namespace=default
time="2021-11-19T07:07:33Z" level=info msg="Removing backup from backup storage" backup=velero-backup-pvs-20211119063547 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:365" name=velero-backup-pvs-20211119063547-dhxjw namespace=default
time="2021-11-19T07:07:33Z" level=info msg="Removing restores" backup=velero-backup-pvs-20211119063547 controller=backup-deletion logSource="pkg/controller/backup_deletion_controller.go:371" name=velero-backup-pvs-20211119063547-dhxjw namespace=default

The log shows, velero is removing PV snapshots.

What did you expect to happen: I exepect to see my PV snapshots is deleted.

Anything else you would like to add: When I check velero code, I can find this pice of code at here:

if backupStore != nil {
		log.Info("Removing PV snapshots")

		if snapshots, err := backupStore.GetBackupVolumeSnapshots(backup.Name); err != nil {
			errs = append(errs, errors.Wrap(err, "error getting backup's volume snapshots").Error())
		} else {
			volumeSnapshotters := make(map[string]velero.VolumeSnapshotter)

			for _, snapshot := range snapshots {
				log.WithField("providerSnapshotID", snapshot.Status.ProviderSnapshotID).Info("Removing snapshot associated with backup")

				volumeSnapshotter, ok := volumeSnapshotters[snapshot.Spec.Location]
				if !ok {
					if volumeSnapshotter, err = volumeSnapshotterForSnapshotLocation(backup.Namespace, snapshot.Spec.Location, c.snapshotLocationLister, pluginManager); err != nil {
						errs = append(errs, err.Error())
						continue
					}
					volumeSnapshotters[snapshot.Spec.Location] = volumeSnapshotter
				}

				if err := volumeSnapshotter.DeleteSnapshot(snapshot.Status.ProviderSnapshotID); err != nil {
					errs = append(errs, errors.Wrapf(err, "error deleting snapshot %s", snapshot.Status.ProviderSnapshotID).Error())
				}
			}
		}
	}

GetBackupVolumeSnapshots this function read the metadata file from S3, the file name looks like fmt.Sprintf("%s-volumesnapshots.json.gz", backup). When I check my metadata on S3, I find this file is empty, and there is another file which name looks like fmt.Sprintf("%s-csi-volumesnapshots.json.gz", backup), the csi file contains the metadata of PV snapshots. And I also find, you have a method to read this file :getCSIVolumeSnapshotKey at here. But you didn’t call this method anywhere.

So, I suspect that you need a condition check for csi PV snapshot and read the correct file content.

Environment:

• Velero version (use velero version): 1.7.0
• Velero features (use velero client config get features): EnableCSI
• Kubernetes version (use kubectl version):1.20.9
• Kubernetes installer & version:
• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

• 👍 for “I would like to see this bug fixed as soon as possible”
• 👎 for “There are more important bugs to focus on right now”