velero: Node-agent high memory usage
What steps did you take and what happened:
Velero is installed with nodeAgent enabled. Backup storage location is configured to use Azure Blob Storage.
apiVersion: velero.io/v1
kind: Restore
metadata:
name: restore-that-uses-a-lot-of-memory
namespace: velero
spec:
backupName: external-backup-1
excludedResources:
- nodes
- events
- events.events.k8s.io
- backups.velero.io
- restores.velero.io
- resticrepositories.velero.io
- csinodes.storage.k8s.io
- volumeattachments.storage.k8s.io
- backuprepositories.velero.io
- policies.rabbitmq.com
existingResourcePolicy: update
hooks: {}
includedNamespaces:
- custom-namespace
itemOperationTimeout: 48h0m0s
uploaderConfig:
parallelFilesDownload: 16
The memory request for node-agent is set to 5Gi. Normally, the limit is also set to 5Gi. To avoid the Pod getting OOMKilled, I have removed the limit for this restore.
What did you expect to happen:
Memory usage stays around, but below, 5Gi.
Anything else you would like to add:
Environment:
- Velero version (use
velero version): v1.15.0 - Velero features (use
velero client config get features): <NOT SET> - Kubernetes version (use
kubectl version): v1.28.2+k3s1 - Kubernetes installer & version: v1.28.2+k3s1
- Cloud provider or hardware configuration: Azure VM
- OS (e.g. from
/etc/os-release): AlmaLinux 9.5 (Teal Serval)
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.
- 👍 for “I would like to see this bug fixed as soon as possible”
- 👎 for “There are more important bugs to focus on right now”
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 4
- Comments: 26 (16 by maintainers)
Commits related to this issue
- Add profiler address parameter on node-agent This allows us to enable the profiler endpoints on both the server and the node agent. This helps me in troubleshooting the high memory usage when restori... — committed to RobKenis/velero by deleted user 2 years ago
- Add profiler address parameter on node-agent This allows us to enable the profiler endpoints on both the server and the node agent. This helps me in troubleshooting the high memory usage when restori... — committed to RobKenis/velero by deleted user 2 years ago
- Add profiler address parameter on node-agent This allows us to enable the profiler endpoints on both the server and the node agent. This helps me in troubleshooting the high memory usage when restori... — committed to RobKenis/velero by deleted user 2 years ago
You may change
AdaptiveThresholdto a smaller value so that the rough estimation could be enabled.That isn’t surprising. Worst case scenario of deduplication-based backup and restore software such as restic and kopia used in Velero is a large number of small files.
Red Hat’s recommendation for a normal sized config is 16GB request, 32GB limit for restic. Your usage requirements are in line with expectations.
Kopia usually uses less resources in Velero 1.15 compared to restic. Will require new backups as kopia and restic repositories are not compatible and have no migration path. You can check which is use by checking the BackupRepository object.