velero: `node-agent` pods won't give back memory after a successful backup
What steps did you take and what happened:
When a backup is scheduled and performed, the node-agent daemonset distributes work on all nodes of the cluster.
The backup takes a lot of memory in our case ( > 6GiB), which is not a problem per-se, when the backup is occurring (Kopia eats lots of memory, that’s expected).
But the issue is that the node-agent processes will keep consuming memory even after all backups are done.
Here is a screenshot of the current memory usage of our node-agent fleet (we have 44 nodes, I cannot fit them all in the screen but all of them are consuming about the same memory). At that moment, there was no backup, restoration or anything else occurring:
This is what happens on our RAM metrics, you can see that the node agents are consuming a very meager amount of RAM, and start eating memory when the backup is running. After the backup ends, the memory usage is still hovering at 180GiB of RAM:
What did you expect to happen:
I expected the RAM usage to go back to close to 0 instead of lingering above 180GiB (for 44 node agents).
The following information will help us better understand what’s going on:
Anything else you would like to add:
Some information on the cluster:
- 44 Nodes, 64GiB of ram for each node
- We have lots of small files to backup for each PVC, which might explain the memory usage
- We don’t use “parallelisation” for the node-agents, there’s only 1 concurrent task
Environment:
-
Velero version (use velero version): 1.14.0 -
Velero features (use velero client config get features): features: -
Kubernetes version (use kubectl version): -
- Client Version: v1.29.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.16
-
-
Kubernetes installer & version: Azure AKS, provisioned by Terraform -
Cloud provider or hardware configuration: Azure -
OS (e.g. from /etc/os-release): Azure Linux distribution
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.
- 👍 for “I would like to see this bug fixed as soon as possible”
- 👎 for “There are more important bugs to focus on right now”
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (10 by maintainers)
FWIW I can reproduce on my personal cluster, where I have very few disks, and I use exclusively the FSB (File system backup) and no Data Mover. This is 60 days of operation with 2 manual restarts of node-agent:
There is only 1 single node, with 1 node-agent. The cluster is a K3S machine and uses velero 1.140 & Kopia to backup longhorn-based filesystems.
It is hard to see, but the agent takes ~40MiB of memory at rest after a fresh reboot, then shoots up to perform the backup, then never goes back to 40MiB. Also, I suspect there’s a small leak after each backup, sufficient to be seen across 10+days. Here’s a zoomed version of the above screen with my notes: