velero: When executing velero backup delete, object data of ${bucket_name}/kopia/${namespace} cannot be deleted from s3 (minio)

As #6916 is closed without real solution I’m opening it again.

Here is some additional information about my config:

Velero server version: 1.13

Backup storage location:

Spec:
  Access Mode:  ReadWrite
  Config:
    Insecure Skip TLS Verify:  true
    Region:                    minio
    s3ForcePathStyle:          true
    s3Url:                     https://my.domain.com:9000
  Default:                     true
  Object Storage:
    Bucket:  velero
  Provider:  aws

Backup storage repository:

Spec:
  Backup Storage Location:  default
  Maintenance Frequency:    1h0m0s
  Repository Type:          kopia
  Restic Identifier:        s3:https://my.domain.com:9000/velero/restic/prometheus
  Volume Namespace:         prometheus

There is a backup schedule job:

NAME              STATUS    CREATED                          SCHEDULE      BACKUP TTL   LAST BACKUP   SELECTOR   PAUSED
bckp-prometheus   Enabled   2024-04-15 15:09:14 +0200 CEST   0 */6 * * *   7h0m0s       2h ago        <none>     false

It creates backup every 6 hours and TTL is 7 hours. That means that at some point I have 2 backups (for 1 hour) and after that period only one is left.

Output of velero backup get is good, there is only one backup (or two if you execute the command one hour after creation of the last backup):

NAME                             STATUS      ERRORS   WARNINGS   CREATED                          EXPIRES   STORAGE LOCATION   SELECTOR
bckp-prometheus-20240419060013   Completed   0        0          2024-04-19 08:00:31 +0200 CEST   4h        default            <none>

This PVC backed up is big around 30GB. After 5 days, Minio showed around 58GB used space and when I entered the bucket, there were files from the first backup. At this time, after 7 days, Minio shows aroung 77GB used space.

Currently, there are about 5300 files in the bucket. Maybe the more than a half are from the first backup. It seems that the first backup lasted for about 15min, because the first file is created at 20:00 and the last one at 20:16.

For this 7 days, there were executed around 26 backups (every 6 hours) and each one (except first one) didn’t last for more than 2min. (difference between first and last created file). These files are also still present in the bucket.

So, the rest 25 backups hold half of the files in the bucket and first one backup other half (rough estimation). It is like the bucket do some kind of versioning (incremental backup). For this bucket, object locking (versioning) is disabled.

When I disable schedule, delete all backups and wait for a few days, Kopia snapshots are still there in the bucket under following path ${bucket_name}/kopia/${namespace}. Everything within ${bucket_name}/backups/ is deleted as expected.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 34 (1 by maintainers)

Most upvoted comments

Yes, but it depends on the arguments you enter. If you set safety=none > $ kopia maintenance run --full --safety=none than it should clean repository immediately. If you run just kopia mainenance run --full Kopia have some mechanism that calulates is it safe to delete all unused objects so you will need to wait some more time

We have the similar situation, but only difference is that we have VM instead of pod. So I guess you can try that way

I did it with Kopia CLI. Firstly, I installed it.

image

Then you need to connect to repository and change repository owner to me as described in messages above.

 kopia repository connect s3 --endpoint name.mydomain.com:{portifneeded} --bucket bucket-name --access-key your-key  --secret-access-key your-secret --disable-tls-verification --prefix kopia/namespace/ --password 'static-passw0rd'

kopia maintenance set --owner=me


kopia maintenance --full