velero: Velero backups partially fail when VolumeSnapshots are deleted during backup run

What steps did you take and what happened: There is an issue in velero when using Kubernetes VolumeSnapshots to backup PVCs. Velero lists all objects to back up at the start of the backup and then tries to back those items up. If there is a velero backup deletion routine (it takes care of retention) running during the backup, it might be that former existing VolumeSnapshots are deleted and can’t be found by the backup routine. This results in partially failed backups.

Our current current workaround is that the velero pods need to be started at a specific time, so it does not clash with the deletion routine.

What did you expect to happen:

Velero should not partially fail due to missing VolumeSnapshot resources. One possible solution might be to ignore non existing resources during the backup.

The following information will help us better understand what’s going on:

time=“2022-07-27T06:54:42Z” level=error msg=“Error backing up item” backup=nine-velero/velero-dailybackup-20220727063705 error=“error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=graphql-stage, name=velero-redis-data-graphql-stage-redis-master-0-8tcvh): rpc error: code = Unknown desc = error getting volume snapshot content from API: volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-344e4bd5-8c72-4473-bd5b-2d396166956e" not found” error.file=“/go/src/github.com/vmware-tanzu/velero/pkg/backup/item_backupper.go:316” error.function=“github.com/vmware-tanzu/velero/pkg/backup.(*itemBackupper).executeActions” logSource=“pkg/backup/backup.go:417” name=velero-redis-data-graphql-stage-redis-master-0-8tcvh

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

  • Velero version (use velero version): v1.10.0
  • Velero features (use velero client config get features): not set
  • Kubernetes version (use kubectl version): v1.22.15-gke.2500
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration: GKE
  • OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 9
  • Comments: 20 (2 by maintainers)

Most upvoted comments

I think ignoring non-existing resources (log warning only) is probably the right solution. If something is deleted before the backup gets to it, then it’s not in the cluster, so it shouldn’t result in errors when it can’t be backed up – i.e. the results should be the same as if it was deleted immediately prior to velero listing the objects to back up.