velero: v1.14.0+: Memory leaks on backups jobs (CSI backups + cloud plugin)

What steps did you take and what happened:

Setup scheduled Backups via CSI driver. No restic, kopia or other file based copy action are in use.

Memory leak.

image

We have an additional to the failed daily schedule a hourly schedule. In grafana, I can see a memory increase after each backup.

What did you expect to happen:

No memory leak

The following information will help us better understand what’s going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

Does work from a pod with in-cluster authentication:

velero debug  --backup opsstack-velero-daily-opsstack-20240625003049 --namespace opsstack
2024/06/25 15:38:09 Collecting velero resources in namespace: opsstack
An error occurred: exec failed: Traceback (most recent call last):
  velero-debug-collector:23:13: in <toplevel>
  <builtin>: in kube_get
Error: could not initialize search client: stat /tmp/.kube/config: no such file or directory

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

  • kubectl logs deployment/velero -n velero
  • velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
  • velero backup logs <backupname>
  • velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
  • velero restore logs <restorename>

velero.zip

Anything else you would like to add:

Environment:

  • Velero version (use velero version): v1.14.0
  • Velero features (use velero client config get features): <NOT SET>
  • Kubernetes version (use kubectl version): v1.29.4
  • Kubernetes installer & version: AKS
  • Cloud provider or hardware configuration: AKS
  • OS (e.g. from /etc/os-release): Azure Linux

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 58 (16 by maintainers)

Most upvoted comments

Some debug progress, but still more to do. We know this happens only for CSI or CSI+datamover backups. I’ve isolated this to the backup finalizer controller. From the added logging, we see in the other controllers we initialize the needed plugins, then at end of reconcile, the relevant plugins are killed. In the finalizer controller case, though it’s a bit different. In my cluster, 3 plugins were initialized (internal, object store, and the OADP openshift BIA/RIA plugin) – then all 3 of these were killed. However, immediately after killing the aws plugin (again this is due to the defer pluginManager.CleanupClients() call at end of finalizer reconcile), a new plugin process starts up. This is triggered by a restartableProcess.ResetIfNeeded() call – I still need to dig into why this is happening at the end of the reconcile. Maybe some embedded defer call somewhere I’m not seeing? I’m not sure yet. I’ll dig into it again tomorrow.

Once #8012 merges we should CP to release-1.14 for 1.14.1.

I think i found the reason. The issue appears to be that each FinalizeBackup call invokes getVolumeInfos to obtain a storage client and refresh volume information. The getVolumeInfos function creates a new pluginManager and uses defer CleanupClients to clean up the process. However, after obtaining the client, the caller proceeds to upload the backup data, during which a new process is created but not cleaned up. Consequently, this newly created process remains persistent.

Which all versions are impacted due to this?

The problem is introduced in https://github.com/vmware-tanzu/velero/pull/7554 , i think all verstions after that commit are impacted.

I think i found the reason. The issue appears to be that each FinalizeBackup call invokes getVolumeInfos to obtain a storage client and refresh volume information. The getVolumeInfos function creates a new pluginManager and uses defer CleanupClients to clean up the process. However, after obtaining the client, the caller proceeds to upload the backup data, during which a new process is created but not cleaned up. Consequently, this newly created process remains persistent.

@jkroepke @sseago hi, i have the same issue, i suspect the codes here introduce the processes leak. https://github.com/vmware-tanzu/velero/blob/a91d2cb036738263d416463d7cc13ac63915d92f/pkg/backup/backup.go#L729-L743 On my machine, I deleted this section of the code, and the issue of leaks did not occur. but, the exact reason why this code was causing a process leak is unclear.

I see. The Azure plugin allocates 100M on startup, so if enough Azure plugin instances pile up, it could cause the OOM.