velero: Restoring a file system backup to a different cluster failed due to Kopia snapshot not found
What steps did you take and what happened:
I am creating a file system backup from a particular namespace in a K8s cluster and restoring it to another cluster. But the Restore is stuck in “In Progress” and it fails after timeout (I am also backing up and restoring the Pod to which the volume is mounted, along with some Secrets and configMaps).
The backup is stored in an S3 bucket and I made sure that the same bucket is linked to the new cluster.
After investigating, I can see that for some reason, the PodVolumeRestore failed with the error:
data path restore failed: Failed to run Kopia restore: Unable to load snapshot 2e97d1c5b03468f979e3143149d46239: snapshot not found
What did you expect to happen: Restore to complete without an issue.
The following information will help us better understand what’s going on:
- The Velero pod and the node agents log erros are the following:
velero-64d44bf455-zcq96 velero time="2024-07-15T09:05:09Z" level=info msg="Found 95 backups in the backup location that do not exist in the cluster and need to be synced" backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:136"
...
velero-64d44bf455-zcq96 velero time="2024-07-15T09:05:09Z" level=info msg="Attempting to sync backup into cluster" backup=school-0000-backup-20240711220015 backupLocation=velero/default controller=backup-sync logSource="pkg/controller/backup_sync_controller.go:144"
....
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:09Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:126"
time="2024-07-15T09:07:11Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:535" restore=velero/school-0000-restore-r6ktt
....
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="No repository found, creating one" backupLocation=default logSource="pkg/repository/ensurer.go:89" repositoryType=kopia volumeNamespace=school-0000
...
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="Initializing backup repository" backupRepo=velero/school-0000-default-kopia-8s97q logSource="pkg/controller/backup_repository_controller.go:216"
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="Set matainenance according to repository suggestion" frequency=1h0m0s logSource="pkg/controller/backup_repository_controller.go:263"
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:11Z" level=info msg="the managed fields for school-0000/ldap-main-0 is patched" logSource="pkg/restore/restore.go:1714" restore=velero/school-0000-restore-r6ktt
....
velero-64d44bf455-zcq96 velero time="2024-07-15T09:07:29Z" level=error msg="unable to successfully complete pod volume restores of pod's volumes" error="pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 2e97d1c5b03468f979e3143149d46239: snapshot not found" logSource="pkg/restore/restore.go:1891" restore=velero/school-0000-restore-r6ktt
- The BackupStorageLocation is Available
Anything else you would like to add: Restoring the backup to the same cluster it was taken from works with no issues, this only happens when I restore to a different cluster.
Environment:
- Velero version (use
velero version):
Client:
Version: v1.13.2
Git commit: -
Server:
Version: v1.13.0
-
Velero features (use
velero client config get features):features: <NOT SET> -
Kubernetes version (use
kubectl version):
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (7 by maintainers)
Answer for all the related problems:
kopia snapshot list --allyou do see snapshots; or when runningkopia content statsyou see the repo is not empty, if may be a switch over problem. However, we don’t expect this would happen and there is no known issue about this. Please collect the log bundle by runningvelero debugon both site when the problem happens so that we can further troubleshoot.