velero: When executing velero backup delete, object data of ${bucket_name}/kopia/${namespace} cannot be deleted from s3 (minio)
As #6916 is closed without real solution I’m opening it again.
Here is some additional information about my config:
Velero server version: 1.13
Backup storage location:
Spec:
Access Mode: ReadWrite
Config:
Insecure Skip TLS Verify: true
Region: minio
s3ForcePathStyle: true
s3Url: https://my.domain.com:9000
Default: true
Object Storage:
Bucket: velero
Provider: aws
Backup storage repository:
Spec:
Backup Storage Location: default
Maintenance Frequency: 1h0m0s
Repository Type: kopia
Restic Identifier: s3:https://my.domain.com:9000/velero/restic/prometheus
Volume Namespace: prometheus
There is a backup schedule job:
NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR PAUSED
bckp-prometheus Enabled 2024-04-15 15:09:14 +0200 CEST 0 */6 * * * 7h0m0s 2h ago <none> false
It creates backup every 6 hours and TTL is 7 hours. That means that at some point I have 2 backups (for 1 hour) and after that period only one is left.
Output of velero backup get is good, there is only one backup (or two if you execute the command one hour after creation of the last backup):
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
bckp-prometheus-20240419060013 Completed 0 0 2024-04-19 08:00:31 +0200 CEST 4h default <none>
This PVC backed up is big around 30GB. After 5 days, Minio showed around 58GB used space and when I entered the bucket, there were files from the first backup. At this time, after 7 days, Minio shows aroung 77GB used space.
Currently, there are about 5300 files in the bucket. Maybe the more than a half are from the first backup. It seems that the first backup lasted for about 15min, because the first file is created at 20:00 and the last one at 20:16.
For this 7 days, there were executed around 26 backups (every 6 hours) and each one (except first one) didn’t last for more than 2min. (difference between first and last created file). These files are also still present in the bucket.
So, the rest 25 backups hold half of the files in the bucket and first one backup other half (rough estimation). It is like the bucket do some kind of versioning (incremental backup). For this bucket, object locking (versioning) is disabled.
When I disable schedule, delete all backups and wait for a few days, Kopia snapshots are still there in the bucket under following path ${bucket_name}/kopia/${namespace}. Everything within ${bucket_name}/backups/ is deleted as expected.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 34 (1 by maintainers)
Yes, but it depends on the arguments you enter. If you set safety=none >
$ kopia maintenance run --full --safety=nonethan it should clean repository immediately. If you run justkopia mainenance run --fullKopia have some mechanism that calulates is it safe to delete all unused objects so you will need to wait some more timeWe have the similar situation, but only difference is that we have VM instead of pod. So I guess you can try that way
I did it with Kopia CLI. Firstly, I installed it.
Then you need to connect to repository and change repository owner to me as described in messages above.