velero: Velero backups fail due to failing to connect to kopia repo: mmap error: cannot allocate memory

What steps did you take and what happened: We have installed Velero version 1.14.0, after upgrading to this version we are facing the below error and the backups are failing for few customers. We are facing this type of issue newly.

"level=error msg=“pod volume backup failed: error to initialize data path: error to boost backup repository connection default–kopia error to connect backup repo: error to connect repo with storage: error to connect to repository: unable to create shared content manager:”

What did you expect to happen: Backups need to be completed successfully

Environment:

  • Velero version (use velero version): v1.14.0
  • Velero features (use velero client config get features): v1.12.3
  • Kubernetes version (use kubectl version): v1.30.1
  • Kubernetes installer & version: v1,30.5-gke.1014003
  • Cloud provider or hardware configuration: GCP
  • OS (e.g. from /etc/os-release): Ubuntu 22.04

velero_pod.log velero_describe.txt velero_backup.log bundle-2024-12-10-07-25-40.zip

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 2
  • Comments: 20 (13 by maintainers)

Most upvoted comments

I have updated the maintenance job logs after allocating the additional memory, and we no longer see the error message

For your current problem, you need to wait more than one completion of repo maintenance. For velero 1.14/kopia 0.17, only full maintenance is effective because there is a bug that quick maintenance won’t do some related tasks. And even though you wait some maintenance and the maintenance starts to fix the current repo problem, it won’t complete, as mentioned above, the memory usage in velero 1.14/kopia 0.17 will be very huge in your case.

Therefore, I suggest you upgrade to 1.15 and wait for several repo maintenance until the problem is fixed.

We did not encounter this type of issue in previous versions

This is because the repo’d just reached this status recently. Since you haven’t run maintenance for a long time, things gradually get worse until the repo cannot tolerate and deny the service. Another clue is that from velero 1.14/kopia 0.17, some index management work has been moved to repo maintenance, so repo maintenance is even more important.

We have allocated more memory to the maintenance job, yet we are still facing partially failed backup issues.

As I mentioned, once this problem happens, it will happen for all repo activities. Because this indicates the repo is in a very bad status, in order to fix the problem, you must run one or more successful full maintenance.