velero: Restoring a file system backup to a different cluster failed due to Kopia snapshot not found

What steps did you take and what happened:

I am creating a file system backup from a particular namespace in a K8s cluster and restoring it to another cluster. But the Restore is stuck in “In Progress” and it fails after timeout (I am also backing up and restoring the Pod to which the volume is mounted, along with some Secrets and configMaps).

The backup is stored in an S3 bucket and I made sure that the same bucket is linked to the new cluster.

After investigating, I can see that for some reason, the PodVolumeRestore failed with the error: data path restore failed: Failed to run Kopia restore: Unable to load snapshot 2e97d1c5b03468f979e3143149d46239: snapshot not found

What did you expect to happen: Restore to complete without an issue.

The following information will help us better understand what’s going on:

  • The Velero pod and the node agents log erros are the following:
velero-64d44bf455-zcq96 velero  time="2024-07-15T09:05:09Z" level=info msg="Found 95 backups in the backup location that do not exist in the cluster and need to be synced" backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:136"
 
...
 
velero-64d44bf455-zcq96 velero  time="2024-07-15T09:05:09Z" level=info msg="Attempting to sync backup into cluster" backup=school-0000-backup-20240711220015 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
 
....
 
velero-64d44bf455-zcq96 velero  time="2024-07-15T09:07:09Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-07-15T09:07:11Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:535" restore=velero/school-0000-restore-r6ktt
 
....
 
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="No repository found, creating one" backupLocation=default logSource="pkg/repository/ensurer.go:89" repositoryType=kopia volumeNamespace=school-0000
 
...
 
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="Initializing backup repository" backupRepo=velero/school-0000-default-kopia-8s97q logSource="pkg/controller/backup_repository_controller.go:216"

velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="Set matainenance according to repository suggestion" frequency=1h0m0s logSource="pkg/controller/backup_repository_controller.go:263"

velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="the managed fields for school-0000/ldap-main-0 is patched" logSource="pkg/restore/restore.go:1714" restore=velero/school-0000-restore-r6ktt
 
....
 
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:29Z" level=error msg="unable to successfully complete pod volume restores of pod's volumes" error="pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 2e97d1c5b03468f979e3143149d46239: snapshot not found" logSource="pkg/restore/restore.go:1891" restore=velero/school-0000-restore-r6ktt
  • The BackupStorageLocation is Available

Anything else you would like to add: Restoring the backup to the same cluster it was taken from works with no issues, this only happens when I restore to a different cluster.

Environment:

  • Velero version (use velero version):
Client:
	Version: v1.13.2
	Git commit: -
Server:
	Version: v1.13.0
  • Velero features (use velero client config get features): features: <NOT SET>

  • Kubernetes version (use kubectl version):

Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (7 by maintainers)

Most upvoted comments

Answer for all the related problems:

  • If you can see kopia repo data, e.g., when running kopia snapshot list --all you do see snapshots; or when running kopia content stats you see the repo is not empty, if may be a switch over problem. However, we don’t expect this would happen and there is no known issue about this. Please collect the log bundle by running velero debug on both site when the problem happens so that we can further troubleshoot.
  • If you cannot see any data in the kopia repo, it may indicates that the source and target sites are not referring to the same object store location or your object store location has been destroyed. You need to double check your env and see what happened.