velero: Velero is not honouring default keep-latest-maintenance-jobs
What steps did you take and what happened:
Deploy velero server with default options(i.e don’t set --keep-latest-maintenance-jobs) and uploader type kopia. Create backups with --snapshot-move-data like below
velero backup create veerendra-backup-8h \
--include-namespaces veerendra-test \
--snapshot-move-data \
--include-resources persistentvolumeclaims \
--csi-snapshot-timeout 8h \
--parallel-files-upload 64
From help, the default value for --keep-latest-maintenance-jobs is 3
$ ./velero server --help | grep keep-latest-maintenance-jobs
--keep-latest-maintenance-jobs int Number of latest maintenance jobs to keep each repository. Optional. (default 3)
When --keep-latest-maintenance-jobs not set, there should be 3 job per backup repository, but
auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185841pp9kg 0/1 Completed 0 12h
auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185877dvtgh 0/1 Completed 0 11h
auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185934zmt2l 0/1 Completed 0 10h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-1718584527745kvm9 0/1 Completed 0 12h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-1718588388245th6g 0/1 Completed 0 11h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-171859205501l2nwj 0/1 Completed 0 10h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171858297hdtlm 0/1 Completed 0 13h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-1718586572jzg8 0/1 Completed 0 12h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859017wg47g 0/1 Completed 0 11h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt 1/1 Running 0 10h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185845006gwgvq 0/1 Completed 0 12h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185883822qls6c 0/1 Completed 0 11h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185935585bjpvd 0/1 Completed 0 10h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-17185844826xwxs5 0/1 Completed 0 12h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-17185880822fz45k 0/1 Completed 0 11h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-171859358179f4c4 0/1 Completed 0 10h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718584507lzxjd 0/1 Completed 0 12h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718588394qpdvr 0/1 Completed 0 11h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718592161gb4h2 0/1 Completed 0 10h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-1718583892q2ns 0/1 Completed 0 12h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-171858778rx9km 0/1 Completed 0 11h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-171859346hd5r9 0/1 Completed 0 10h
repo-maintain-job-1718583275574-fjvdx 0/1 Completed 0 13h
repo-maintain-job-1718583575574-9dxbj 0/1 Completed 0 12h
repo-maintain-job-1718583587497-59dzn 0/1 Completed 0 12h
repo-maintain-job-1718583595539-c49bk 0/1 Completed 0 12h
repo-maintain-job-1718583875575-xsdq4 0/1 Completed 0 12h
repo-maintain-job-1718583881845-9jwsm 0/1 Completed 0 12h
repo-maintain-job-1718583886866-btw8n 0/1 Completed 0 12h
repo-maintain-job-1718584475577-ms46g 0/1 Completed 0 12h
repo-maintain-job-1718584488615-ktw7z 0/1 Completed 0 12h
repo-maintain-job-1718584494633-5sjqv 0/1 Completed 0 12h
repo-maintain-job-1718584513718-vkh4h 0/1 Completed 0 12h
repo-maintain-job-1718584533746-6gq8g 0/1 Completed 0 12h
repo-maintain-job-1718584540761-ccmv2 0/1 Completed 0 12h
repo-maintain-job-1718586875584-g4d6q 0/1 Completed 0 12h
repo-maintain-job-1718587175585-8jm8l 0/1 Completed 0 11h
repo-maintain-job-1718587475585-tzsnf 0/1 Completed 0 11h
repo-maintain-job-1718587483390-rvfpx 0/1 Completed 0 11h
repo-maintain-job-1718587489403-bpfkg 0/1 Completed 0 11h
repo-maintain-job-1718587496427-mk87t 0/1 Completed 0 11h
repo-maintain-job-1718587775585-whwg8 0/1 Completed 0 11h
repo-maintain-job-1718588075586-7l2l9 0/1 Completed 0 11h
repo-maintain-job-1718588097309-kgjw2 0/1 Completed 0 11h
repo-maintain-job-1718588375587-8ddvp 0/1 Completed 0 11h
repo-maintain-job-1718588400297-bp8bb 0/1 Completed 0 11h
repo-maintain-job-1718588415359-6g2sw 0/1 Completed 0 11h
repo-maintain-job-1718588423385-lkddt 0/1 Completed 0 11h
repo-maintain-job-1718590475592-9wdgv 0/1 Completed 0 11h
repo-maintain-job-1718590775593-kls5g 0/1 Completed 0 10h
repo-maintain-job-1718592174416-4298z 0/1 Completed 0 10h
repo-maintain-job-1718593379857-kz96d 0/1 Completed 0 10h
repo-maintain-job-1718593386910-6fsdg 0/1 Completed 0 10h
repo-maintain-job-1718593396937-55vpc 0/1 Completed 0 10h
repo-maintain-job-1718593435141-l4kmt 0/1 Completed 0 10h
repo-maintain-job-1718593443182-wjv6t 0/1 Completed 0 10h
repo-maintain-job-1718593475326-ftj4r 0/1 Completed 0 10h
repo-maintain-job-1718593548635-w8m5t 0/1 Completed 0 10h
repo-maintain-job-1718593573752-b9xb4 0/1 Completed 0 10h
repo-maintain-job-1718593587806-nr2sc 0/1 Completed 0 10h
repo-maintain-job-1718593596321-87w5k 0/1 Completed 0 10h
search-backup-prod-de1-kopia-52bbc-maintain-job-1718585675pbgsm 0/1 Completed 0 12h
search-backup-prod-de1-kopia-52bbc-maintain-job-17185892756kpff 0/1 Completed 0 11h
search-backup-prod-de1-kopia-52bbc-maintain-job-1718593601bh4vp 0/1 Completed 0 10h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-171858625mdpz 0/1 Completed 0 12h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-17185898g2z2p 0/1 Completed 0 11h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-17185935dsdnl 0/1 Completed 0 10h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171858452rqqhf 0/1 Completed 0 12h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171858840dmvvc 0/1 Completed 0 11h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171859337m9jld 0/1 Completed 0 10h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-171858597558c469z 0/1 Completed 0 12h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-171858957558rtlkb 0/1 Completed 0 11h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-1718593609917bkf7 0/1 Completed 0 10h
What did you expect to happen:
There should be only 3 maintain-job pods per backup repositories
The following information will help us better understand what’s going on:
If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help
If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs deployment/velero -n velerovelero backup describe <backupname>orkubectl get backup/<backupname> -n velero -o yamlvelero backup logs <backupname>velero restore describe <restorename>orkubectl get restore/<restorename> -n velero -o yamlvelero restore logs <restorename>
Anything else you would like to add:
velero-server logs -> https://gist.github.com/veerendra2/a25ba2e55070230af7cf791350f327f8
Environment:
velero-server
command:
- /velero
args:
- server
### Flags
- --uploader-type=kopia
- --default-backup-storage-location=backup
- --log-format=json
- --keep-latest-maintenance-jobs=1
### Global Flags
- --features=EnableCSI
- --default-backup-ttl=${velero_schedule_ttl}
- --resource-timeout=60m
- --terminating-resource-timeout=60m
node-agent
containers:
- args:
- node-agent
- server
- --data-mover-prepare-timeout=60m
- --resource-timeout=60m
- --features=EnableCSI
- --log-format=json
command:
- /velero
- Velero version (use
velero version): Serverv1.13.2 - Velero features (use
velero client config get features): - Kubernetes version (use
kubectl version):Server Version: v1.28.5 - Kubernetes installer & version:
- Cloud provider or hardware configuration: Azure AKS
- OS (e.g. from
/etc/os-release): Ubuntu
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.
- 👍 for “I would like to see this bug fixed as soon as possible”
- 👎 for “There are more important bugs to focus on right now”
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (8 by maintainers)
Please share the velero log bundle from the time when the maintenance jobs were created to the latest.