velero: Error: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
What steps did you take and what happened:
There are bunch of clusters where the backup of snapshots are failing with the following error:
**What did you expect to happen: ** Cluster seems to be healthy; we are not seeing any issue related to pod crunch or Api server in the timeframe when this error occurred.
The following information will help us better understand what’s going on: Found a related GitHub issues: https://github.com/helm/helm/issues/12154 Error is getting thrown at this Code Path: https://github.com/kubernetes/client-go/blob/354ed1bc9f1f48c820fdc5e84b566acfd716cf42/rest/request.go#L619
If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help
If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs deployment/velero -n velerovelero backup describe <backupname>orkubectl get backup/<backupname> -n velero -o yamlvelero backup logs <backupname>velero restore describe <restorename>orkubectl get restore/<restorename> -n velero -o yamlvelero restore logs <restorename>
Anything else you would like to add:
Environment:
- Velero version (use
velero version): - Velero features (use
velero client config get features): - Kubernetes version (use
kubectl version): - Kubernetes installer & version:
- Cloud provider or hardware configuration:
- OS (e.g. from
/etc/os-release):
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.
- 👍 for “I would like to see this bug fixed as soon as possible”
- 👎 for “There are more important bugs to focus on right now”
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 9
- Comments: 18 (9 by maintainers)
Hi, any update on this issue? I seem to be dealing with the same issue on v1.14.0:
I think these errors and warnings are not related the original issue. This issue reported the backup failed with the client exceed context deadine.
Those warnings and errors are reported by running the
velero backup describeCLI. The reason is the Velero CLI cannot connect to the backup storage’s OSS bucket.Please check the CLI issue was caused by network flappy issue. If it’s not, please make sure the place running Velero CLI can connect the OSS bucket API.
@blackpiglet @curtis-baillie did you find a solution for this:
Still happening in v1.15.2. The backup is successful with no errors or warnings. the restore is also successful but it shows the above warning and error messages. I am using NetApp StorageGRID as backup location if this is relevant.
Sorry, we already removed Velero now and do our volume snapshots by an own controller.