velero: Azure disk snapshot 429 ratelimit with velero-plugin-for-microsoft-azure
We have a couple of AKS clusters in Azure and we enabled the CSI feature to snapshot/backup PVCs. It’s working fine for the clusters with fewer PVCs but for a cluster with 65 PVCs azure starts sending a 429 response code.
The issue is that whenever the snapshoter tries to take a snapshot or fetch the existing backup it tries to use Azure API to list the storage account keys and It exhausts Azure API.
This is the error code:
failed with storage.FileSharesClient#Get: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code=\\\"TooManyRequests\\\" Message=\\\"The request is being throttled as the limit has been reached for operation type - Read_ObservationWindow_00:05:00. For more information, see - https://aka.ms/srpthrottlinglimits\\\", accountName: \\\"<storageaccountname>\\\"\"" backup=<veleronamespace>/<veleroPod>cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:259" pluginName=velero-plugin-for-csi
and this is the Azure activity on the storage account which is currently throttled.
A possible fix for the issue: caching the storage account key instead of listing keys each time for each request
- Velero version: 1.13.0
- Velero features: EnableCSI
- Kubernetes version: v1.27.3
- Cloud provider or hardware configuration: Azure AKS
- OS : azure linux
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 16 (6 by maintainers)
Thank you @mayankagg9722 for helping with this!
Hi! In my case, I applied the indicated configuration and I have not had warnings or errors in my aks backup again. the errors are 2 orphaned disks from some test in the development cluster . thank you