velero: Restore: Kopia failed to change file's mod time
What steps did you take and what happened:
- Install and configure csi-s3 driver e.g.
csi-s3storage class kubectl create pvc ...Deploy Deployment and PVC withcsi-s3storage class. There is a demo manifestvelero backup create ...Create a restore using kopia(FSB) for that PVC datakubectl delete ...Delete Deployment and PVC to simulate disastervelero restoreRestore resources and volume data from backupvelero restore describe --details <restore-name>:RestoreandPodVolumeRestorefailedvelero restore logs <restore-name> | grep level=error
time="2024-10-26T15:21:31Z" level=error msg="unable to successfully complete pod volume restores of pod's volumes" error="pod volume restore failed: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: finish directory: error setting attributes: could not change mod time on /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: chtimes /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: operation not supported" logSource="pkg/restore/restore.go:1881" restore=velero/fsb-pvc-restore4
time="2024-10-26T15:21:32Z" level=error msg="Velero restore error: pod volume restore failed: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: finish directory: error setting attributes: could not change mod time on /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: chtimes /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: operation not supported" logSource="pkg/controller/restore_controller.go:584" restore=velero/fsb-pvc-restore4
What did you expect to happen:
Provide argument to skip changing mod time for kopia even if restic
The following information will help us better understand what’s going on: bundle-2024-10-27-00-54-23.tar.gz
Anything else you would like to add:
K8s resources and volume data are restored successfully, .
But Restore and PodVolumeRestore failed in almost the final step of kopia restoration. Logs shows kopia failed to change mod time of csi-s3 PV’s files.
The csi-s3 driver uses GeeseFS to mount the volume. I guess it is due to GeeseFS not supporting this operation.
🟢 > velero restore describe --details fsb-pvc-restore4
Name: fsb-pvc-restore4
Namespace: velero
Labels: <none>
Annotations: <none>
Phase: PartiallyFailed (run 'velero restore logs fsb-pvc-restore4' for more information)
Total items to be restored: 10
Items restored: 10
Started: 2024-10-27 00:20:58 +0900 JST
Completed: 2024-10-27 00:21:35 +0900 JST
Warnings:
Velero: <none>
Cluster: could not restore, CustomResourceDefinition "ciliumendpoints.cilium.io" already exists. Warning: the in-cluster version is different than the backed-up version
Namespaces:
fsb-pvc-ns: could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
Errors:
Velero: pod volume restore failed: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: finish directory: error setting attributes: could not change mod time on /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: chtimes /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: operation not supported
Cluster: <none>
Namespaces: <none>
Backup: fsb-pvc-backup4
Namespaces:
Included: all namespaces found in the backup
Excluded: <none>
Resources:
Included: *
Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
Cluster-scoped: auto
Namespace mappings: <none>
Label selector: <none>
Or label selector: <none>
Restore PVs: auto
kopia Restores:
Failed:
fsb-pvc-ns/sleep-55d4846d9d-xclbk: large-pvc
CSI Snapshot Restores: <none included>
Existing Resource Policy: <none>
ItemOperationTimeout: 4h0m0s
Preserve Service NodePorts: auto
Uploader config:
HooksAttempted: 0
HooksFailed: 0
Resource List:
apiextensions.k8s.io/v1/CustomResourceDefinition:
- ciliumendpoints.cilium.io(failed)
apps/v1/Deployment:
- fsb-pvc-ns/sleep(created)
apps/v1/ReplicaSet:
- fsb-pvc-ns/sleep-55d4846d9d(created)
cilium.io/v2/CiliumEndpoint:
- fsb-pvc-ns/sleep-55d4846d9d-xclbk(created)
v1/ConfigMap:
- fsb-pvc-ns/kube-root-ca.crt(failed)
v1/Namespace:
- fsb-pvc-ns(created)
v1/PersistentVolume:
- pvc-e9a908e9-6458-43f6-9a52-844d5a384ca4(skipped)
v1/PersistentVolumeClaim:
- fsb-pvc-ns/large-pvc(created)
v1/Pod:
- fsb-pvc-ns/sleep-55d4846d9d-xclbk(created)
v1/ServiceAccount:
- fsb-pvc-ns/default(skipped)
Environment:
- Velero version (use
velero version): v1.14.1 - Velero features (use
velero client config get features): NOT SET - Kubernetes version (use
kubectl version): v1.28.14 - Kubernetes installer & version: k3s v1.28.14+k3s1
- Cloud provider or hardware configuration: csi-s3 driver with Cloudflare R2 storage
- OS (e.g. from
/etc/os-release): Debian 11
# Client side executions
🟢 > velero version
Client:
Version: v1.14.1
Git commit: 8afe3cea8b7058f7baaf447b9fb407312c40d2da
Server:
Version: v1.14.1
🟢 > velero client config get features
features: <NOT SET>
🟢 > kubectl version
Client Version: v1.28.14
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.14+k3s1
# Cluster side execution
root:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.
- 👍 for “I would like to see this bug fixed as soon as possible”
- 👎 for “There are more important bugs to focus on right now”
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 23 (9 by maintainers)
To fully support GeeseFS in the mentioned CSI driver, it might also be necessary to set
SkipPermissionsandSkipOwnersbut we won’t likely be verifying this for the user, user just have to do their own testing. We are simply exposing more kopia options.It will be resolved when the PR to kopia is merged and kopia releases a patch with the fix.
At this moment I don’t see we have to target it to a specific version of velero.
Hi @gdm257 @reasonerjt @Lyndon-Li
I am also facing similar issue, unable to restore the backup of file system backup, getting below error.
Message: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: copy directory contents: finish directory: error setting attributes: could not change mod time on /host_pods/d6b24b1e-b057-4ab3-a64d-be41a8123682/volumes/kubernetes.io~csi/csi-fss-b609854b-aa87-4757-965c-8279bfea10c2/mount/.snapshot: chtimes /host_pods/d6b24b1e-b057-4ab3-a64d-be41a8123682/volumes/kubernetes.io~csi/csi-fss-b609854b-aa87-4757-965c-8279bfea10c2/mount/.snapshot: read-only file system
Working on velero appVersion: 1.14.1 also tried with new version 1.15.1, getting same error. anybody has clue to resolve this issue.
please help on this. Thanks in advance !