velero: Restore: Kopia failed to change file's mod time

What steps did you take and what happened:

  1. Install and configure csi-s3 driver e.g. csi-s3 storage class
  2. kubectl create pvc ... Deploy Deployment and PVC with csi-s3 storage class. There is a demo manifest
  3. velero backup create ... Create a restore using kopia(FSB) for that PVC data
  4. kubectl delete ... Delete Deployment and PVC to simulate disaster
  5. velero restore Restore resources and volume data from backup
  6. velero restore describe --details <restore-name>: Restore and PodVolumeRestore failed
  7. velero restore logs <restore-name> | grep level=error
time="2024-10-26T15:21:31Z" level=error msg="unable to successfully complete pod volume restores of pod's volumes" error="pod volume restore failed: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: finish directory: error setting attributes: could not change mod time on /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: chtimes /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: operation not supported" logSource="pkg/restore/restore.go:1881" restore=velero/fsb-pvc-restore4
time="2024-10-26T15:21:32Z" level=error msg="Velero restore error: pod volume restore failed: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: finish directory: error setting attributes: could not change mod time on /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: chtimes /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: operation not supported" logSource="pkg/controller/restore_controller.go:584" restore=velero/fsb-pvc-restore4

What did you expect to happen:

Provide argument to skip changing mod time for kopia even if restic

The following information will help us better understand what’s going on: bundle-2024-10-27-00-54-23.tar.gz

Anything else you would like to add:

K8s resources and volume data are restored successfully, .

But Restore and PodVolumeRestore failed in almost the final step of kopia restoration. Logs shows kopia failed to change mod time of csi-s3 PV’s files.

The csi-s3 driver uses GeeseFS to mount the volume. I guess it is due to GeeseFS not supporting this operation.

🟢 > velero restore describe --details fsb-pvc-restore4
Name:         fsb-pvc-restore4
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       PartiallyFailed (run 'velero restore logs fsb-pvc-restore4' for more information)
Total items to be restored:  10
Items restored:              10

Started:    2024-10-27 00:20:58 +0900 JST
Completed:  2024-10-27 00:21:35 +0900 JST

Warnings:
  Velero:     <none>
  Cluster:  could not restore, CustomResourceDefinition "ciliumendpoints.cilium.io" already exists. Warning: the in-cluster version is different than the backed-up version
  Namespaces:
    fsb-pvc-ns:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version

Errors:
  Velero:   pod volume restore failed: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: finish directory: error setting attributes: could not change mod time on /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: chtimes /host_pods/89f5c9ea-f68d-485a-9beb-f317b551722c/volumes/kubernetes.io~csi/pvc-70e4ad4e-fb92-4100-91ab-40da1aeddfb7/mount: operation not supported
  Cluster:    <none>
  Namespaces: <none>

Backup:  fsb-pvc-backup4

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Or label selector:  <none>

Restore PVs:  auto

kopia Restores:
  Failed:
    fsb-pvc-ns/sleep-55d4846d9d-xclbk: large-pvc

CSI Snapshot Restores: <none included>

Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s

Preserve Service NodePorts:  auto

Uploader config:


HooksAttempted:   0
HooksFailed:      0

Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - ciliumendpoints.cilium.io(failed)
  apps/v1/Deployment:
    - fsb-pvc-ns/sleep(created)
  apps/v1/ReplicaSet:
    - fsb-pvc-ns/sleep-55d4846d9d(created)
  cilium.io/v2/CiliumEndpoint:
    - fsb-pvc-ns/sleep-55d4846d9d-xclbk(created)
  v1/ConfigMap:
    - fsb-pvc-ns/kube-root-ca.crt(failed)
  v1/Namespace:
    - fsb-pvc-ns(created)
  v1/PersistentVolume:
    - pvc-e9a908e9-6458-43f6-9a52-844d5a384ca4(skipped)
  v1/PersistentVolumeClaim:
    - fsb-pvc-ns/large-pvc(created)
  v1/Pod:
    - fsb-pvc-ns/sleep-55d4846d9d-xclbk(created)
  v1/ServiceAccount:
    - fsb-pvc-ns/default(skipped)

Environment:

  • Velero version (use velero version): v1.14.1
  • Velero features (use velero client config get features): NOT SET
  • Kubernetes version (use kubectl version): v1.28.14
  • Kubernetes installer & version: k3s v1.28.14+k3s1
  • Cloud provider or hardware configuration: csi-s3 driver with Cloudflare R2 storage
  • OS (e.g. from /etc/os-release): Debian 11
# Client side executions
🟢 > velero version
Client:
        Version: v1.14.1
        Git commit: 8afe3cea8b7058f7baaf447b9fb407312c40d2da
Server:
        Version: v1.14.1

🟢 > velero client config get features
features: <NOT SET>

🟢 > kubectl version
Client Version: v1.28.14
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.14+k3s1
# Cluster side execution
root:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 23 (9 by maintainers)

Most upvoted comments

To fully support GeeseFS in the mentioned CSI driver, it might also be necessary to set SkipPermissions and SkipOwners but we won’t likely be verifying this for the user, user just have to do their own testing. We are simply exposing more kopia options.

Does not store file permissions and custom modification times

It will be resolved when the PR to kopia is merged and kopia releases a patch with the fix.

At this moment I don’t see we have to target it to a specific version of velero.

Hi @gdm257 @reasonerjt @Lyndon-Li

I am also facing similar issue, unable to restore the backup of file system backup, getting below error.

Message: data path restore failed: Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: copy directory contents: finish directory: error setting attributes: could not change mod time on /host_pods/d6b24b1e-b057-4ab3-a64d-be41a8123682/volumes/kubernetes.io~csi/csi-fss-b609854b-aa87-4757-965c-8279bfea10c2/mount/.snapshot: chtimes /host_pods/d6b24b1e-b057-4ab3-a64d-be41a8123682/volumes/kubernetes.io~csi/csi-fss-b609854b-aa87-4757-965c-8279bfea10c2/mount/.snapshot: read-only file system

Working on velero appVersion: 1.14.1 also tried with new version 1.15.1, getting same error. anybody has clue to resolve this issue.

please help on this. Thanks in advance !