velero: Azure disk snapshot 429 ratelimit with velero-plugin-for-microsoft-azure

We have a couple of AKS clusters in Azure and we enabled the CSI feature to snapshot/backup PVCs. It’s working fine for the clusters with fewer PVCs but for a cluster with 65 PVCs azure starts sending a 429 response code.

The issue is that whenever the snapshoter tries to take a snapshot or fetch the existing backup it tries to use Azure API to list the storage account keys and It exhausts Azure API.

This is the error code:

 failed with storage.FileSharesClient#Get: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code=\\\"TooManyRequests\\\" Message=\\\"The request is being throttled as the limit has been reached for operation type - Read_ObservationWindow_00:05:00. For more information, see - https://aka.ms/srpthrottlinglimits\\\", accountName: \\\"<storageaccountname>\\\"\"" backup=<veleronamespace>/<veleroPod>cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:259" pluginName=velero-plugin-for-csi

and this is the Azure activity on the storage account which is currently throttled. Screenshot 2024-02-06 105147

A possible fix for the issue: caching the storage account key instead of listing keys each time for each request

  • Velero version: 1.13.0
  • Velero features: EnableCSI
  • Kubernetes version: v1.27.3
  • Cloud provider or hardware configuration: Azure AKS
  • OS : azure linux

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 16 (6 by maintainers)

Most upvoted comments

Thank you @mayankagg9722 for helping with this!

Hello @behroozam @jfmulero this is a already known issue and for this you have to add useDataPlane: true in your volume snapshot class so that it will use data plane sdk for storage which doesn’t have any throttling limits as such and this will solve your concern.

Ref: https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/driver-parameters.md

kubernetes-sigs/azurefile-csi-driver#1687

Azure/AKS#804

NAME                                STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
velero-devschedule-20250204010006   PartiallyFailed   2        0          2025-02-04 01:00:06 +0000 UTC   4d        veleroaksdev       <none>
velero-devschedule-20250203010033   PartiallyFailed   20       1218       2025-02-03 01:00:33 +0000 UTC   3d        veleroaksdev       <none>
velero-devschedule-20250202010032   PartiallyFailed   20       1200       2025-02-02 01:00:32 +0000 UTC   2d        veleroaksdev       <none>
velero-devschedule-20250201010031   PartiallyFailed   20       1217       2025-02-01 01:00:31 +0000 UTC   1d        veleroaksdev       <none>
velero-devschedule-20250131010028   PartiallyFailed   18       1247       2025-01-31 01:00:29 +0000 UTC   16h       veleroaksdev       <none>

Hi! In my case, I applied the indicated configuration and I have not had warnings or errors in my aks backup again. the errors are 2 orphaned disks from some test in the development cluster . thank you