velero: Permission Denied during Restore using restic

What steps did you take and what happened:

Created backup of my resources with PV on cluster 1
Trying to restore it on cluster 2 Storage used : Minio (both cluster pointing to same backup storage location)
Restore completes without any errors (Deployments, PV can be seen on cluster 2)
But the deployments never come up in running state
- pod associated to it is in pending state
- docker container of pod throwing error ERROR looking for /restores/mongodb-app-ibm-mongodb-dev-datavolume/.velero/7817a7da-76ef-11e9-8851-fa163e586ef1: stat /restores/mongodb-app-ibm-mongodb-dev-datavolume/.velero/7817a7da-76ef-11e9-8851-fa163e586ef1: permission denied

What did you expect to happen: After restore is complete, expected all deployments to be in up and running state.

The output of the following commands will help us better understand what’s going on: (Pasting long output into a GitHub gist or other pastebin is fine.)

velero backup logs <backupname> time="2019-05-15T06:56:13Z" level=info msg="Backup completed successfully" backup=velero/mongodb-app-with-pv-1 logSource="pkg/backup/backup.go:295"
velero restore logs <restorename> time="2019-05-15T08:58:23Z" level=info msg="restore completed" backup=mongodb-app-with-pv-1 logSource="pkg/controller/restore_controller.go:495" restore=velero/mongodb-app-with-pv-1-20190515085729

Anything else you would like to add:

Restore of resources without PV associated with them works fine.
In above scenario if i scale my deployments to 0 and re-scale to 1 again the deployment comes up and is running there after.

Environment:

Velero version (use velero version): v0.11.0
Kubernetes version (use kubectl version): v1.12.4
Kubernetes installer & version: ICP 3.1.2
Cloud provider or hardware configuration: HDC-Hursley
OS (e.g. from /etc/os-release): Ubuntu 16.04

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 27 (13 by maintainers)

Most upvoted comments

I’m running into what seems to be the same problem. Please let me know if I should open a new issue though.

After a velero restore of a PVC, pod and deployment, the restic-wait container hangs with

ERROR looking for /restores/database/.velero/5f889bf8-9591-431f-9486-81b20f42ebee: stat /restores/database/.velero/5f889bf8-9591-431f-9486-8
1b20f42ebee: permission denied

Permissions of the contents of /restores/ are

drwx------ 20 999 root 4096 Jul 29 15:56 database

while the output of id is

uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup)

There’s no psp I think. I can try changing the pod’s security context, but I’m not sure if that’s possible without disturbing the regular pod containers, and ideally such intervention would not be necessary at all. Perhaps the init container added by velero should have a securityContext override so it will work regardless of the pod settings?

ariep on Aug 5, 2021