velero: failed to wait PVBs - 1.15.1 - Backup

What steps did you take and what happened: Any creation of backup Manually or with Schedule.

Errors: Velero: message: /failed to wait PVBs processed for the ItemBlock error: /failed to list PVBs: the server was unable to return a response in the time allotted, but may still be processing the request (get podvolumebackups.velero.io)

What did you expect to happen: a Completed status and not PartiallyFailed.

Anything else you would like to add:

Environment:

  • Velero version (use velero version):
Client:
        Version: v1.15.1
        Git commit: 32499fc287815058802c1bc46ef620799cca7392
Server:
        Version: v1.15.1
  • Velero features (use velero client config get features): features: <NOT SET>
  • Kubernetes version (use kubectl version):
  • Kubernetes installer & version:
Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.1
  • Cloud provider or hardware configuration: OVH Managed Kubernetes, Bucket S3 OVH and Scaleway

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 3
  • Comments: 22 (8 by maintainers)

Commits related to this issue

Most upvoted comments

@Gui13 What error did you get? Is it the same with this one failed to list PVBs: the server was unable to return a response in the time allotted, but may still be processing the request (get podvolumebackups.velero.io)?

I ran 3 backups at 5 mins intervals

$ velero backup get
NAME                                       STATUS       ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION     SELECTOR
ovh-daily-csi-snapshot-20250109162052      Completed    0        0          2025-01-09 17:20:53 +0100 CET   3d        ovh-s3-bucket        <none>

scaleway-daily-fsb-backup-20250109162248   Completed         0        13         2025-01-09 17:25:03 +0100 CET   19d       scaleway-s3-bucket   <none>

ovh-daily-fsb-backup-20250109162145        PartiallyFailed   26       13         2025-01-09 17:22:45 +0100 CET   19d       ovh-s3-bucket        <none>

type of Failure ( * 26 times)=

             name: /velero-redis-data-verifybadge-redis-master-0-h8sqc message: /Error backing up item error: /error executing custom action (groupResource=volumesnapshots.snapshot.storage.k8s.io, namespace=pqp-verifybadge, name=velero-redis-data-verifybadge-redis-master-0-h8sqc): rpc error: code = Unknown desc = volumesnapshots.snapshot.storage.k8s.io "velero-redis-data-verifybadge-redis-master-0-h8sqc" not found

Then i just relaunch the PartillayFailed backup and :

ovh-daily-fsb-backup-20250109162935 Completed 0 13 2025-01-09 17:29:36 +0100 CET 19d ovh-s3-bucket <none>

I don’t know what happened, but it seems there are no PVB errors now with your fix. 👍

I will try it now and give some feedbacks.

Hi,

we have the same problem. The problem occurs after updating from velero 1.14 to velero 1.15.1. Interesting is that since the update at the backup time the Kubernetes API memory spikes extremely upwards.

Screenshot 2025-01-09 at 14 11 50

Hi @fwernert, I made some approvement to avoid the the server was unable to return a response in the time allotted error when there are a lot of PVBs, appreciate if you can try that in your env to verify whether it works or not. Here is the Velero image: yinw/velero:verification01.