velero: Backup partially failed with csi plugin 0.6.0-rc2 on OVH cluster


name: Bug report about: Using the velero 1.12.0 Data Movement feature on OVH managed cluster make backup partially failed while using matching csi plugin version v0.6.0-rc2 while it was working on v0.5.1.


What steps did you take and what happened: I wantend to test the Data Movement feature. I installed velero CLI v1.12.0-rc.2

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0-rc.2,velero/velero-plugin-for-csi:v0.6.0-rc.2 \
  --no-default-backup-location \
  --features=EnableCSI \
  --no-secret \
  --use-node-agent

# Create kubernetes secret with s3 credentials

# Create velero storage location
velero backup-location create --bucket "${OVH_CLOUD_PROJECT_SERVICE}-my-cluster-backup" --provider aws --config region=gra,s3ForcePathStyle="true",s3Url=https://s3.gra.io.cloud.ovh.net "my-cluster-backup" --credential "my-cluster-backup=cloud"

# Create velero snapshot location
velero snapshot-location create --provider aws --config region=gra,s3ForcePathStyle="true",s3Url=https://s3.gra.io.cloud.ovh.net "my-cluster-backup" --credential "my-cluster-backup=cloud"

# VolumeSnapshotClass for ovh

# Create the backup
velero backup create "my-cluster-backup-${uuid}" --snapshot-move-data --storage-location "my-cluster-backup" --volume-snapshot-locations "my-cluster-backup" --csi-snapshot-timeout 10m

The backup ended in a PartiallyFailed state with error for the majority of PVC: Fail to wait VolumeSnapshot snapshot handle created. Still some PVC was able to be backup while some didn’t, so I am guessing it’s realated to some timeout error.

What did you expect to happen:

I expected the backup to work in the rc version of the csi plugin since nothing else changed on the cluster except this version.

The following information will help us better understand what’s going on:

The bundle extract from velero debug --backup:

bundle-2023-09-21-11-15-47.tar.gz

Anything else you would like to add:

I tried running a backup with the exact same install commands mentionned before but changing the plugins version of the csi plugin to v0.5.1:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0-rc.2,velero/velero-plugin-for-csi:v0.5.1 \
  --no-default-backup-location \
  --features=EnableCSI \
  --no-secret \
  --use-node-agent

And it’s worked without any error. Here is the debug bundle of the working backup with csi plugin in version v0.5.1. bundle-2023-09-21-12-20-13.tar.gz

Of course even if it’s worked it is missing the DataUpload part to achieve DataMovement so it is not what I am looking for.

Environment:

  • Velero version (use velero version):v1.12.0-rc.2 7112c62e493b0f7570f0e7cd2088f8cad968db99
  • Velero features (use velero client config get features):
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-14T09:53:42Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.7", GitCommit:"84e1fc493a47446df2e155e70fca768d2653a398", GitTreeState:"clean", BuildDate:"2023-07-19T12:16:45Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes installer & version: OVH v1.26.7
  • Cloud provider or hardware configuration: OVH
  • OS (e.g. from /etc/os-release): - RuntimeOS: linux - RuntimeArch: amd64

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 18
  • Comments: 15 (7 by maintainers)

Most upvoted comments

I took some time to debug while looking at the source code so here are my investigations if it can help in any way:

But if it is iterating two time on this loop it would mean that the first time it was able to successfuly get the VolumeSnapshot and reached the Waiting log line… At this point I don’t have any more idea so I hope a maintainer can help me 👼 .