velero: Velero is not honouring default keep-latest-maintenance-jobs

What steps did you take and what happened:

Deploy velero server with default options(i.e don’t set --keep-latest-maintenance-jobs) and uploader type kopia. Create backups with --snapshot-move-data like below

velero backup create veerendra-backup-8h \
  --include-namespaces veerendra-test \
  --snapshot-move-data \
  --include-resources persistentvolumeclaims \
  --csi-snapshot-timeout 8h \
  --parallel-files-upload 64

From help, the default value for --keep-latest-maintenance-jobs is 3

$ ./velero server --help | grep keep-latest-maintenance-jobs
      --keep-latest-maintenance-jobs int                    Number of latest maintenance jobs to keep each repository. Optional. (default 3)

When --keep-latest-maintenance-jobs not set, there should be 3 job per backup repository, but

auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185841pp9kg   0/1     Completed   0                12h
auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185877dvtgh   0/1     Completed   0                11h
auditlog-backup-prod-de1-kopia-xmh2b-maintain-job-17185934zmt2l   0/1     Completed   0                10h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-1718584527745kvm9   0/1     Completed   0                12h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-1718588388245th6g   0/1     Completed   0                11h
auth-backup-prod-de1-kopia-dhjm4-maintain-job-171859205501l2nwj   0/1     Completed   0                10h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171858297hdtlm   0/1     Completed   0                13h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-1718586572jzg8   0/1     Completed   0                12h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859017wg47g   0/1     Completed   0                11h
backend-backup-prod-de1-kopia-7jxj2-maintain-job-171859377kblkt   1/1     Running     0                10h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185845006gwgvq   0/1     Completed   0                12h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185883822qls6c   0/1     Completed   0                11h
email-backup-prod-de1-kopia-84jqf-maintain-job-17185935585bjpvd   0/1     Completed   0                10h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-17185844826xwxs5   0/1     Completed   0                12h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-17185880822fz45k   0/1     Completed   0                11h
notes-backup-prod-de1-kopia-5jx9m-maintain-job-171859358179f4c4   0/1     Completed   0                10h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718584507lzxjd   0/1     Completed   0                12h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718588394qpdvr   0/1     Completed   0                11h
pacman-backup-prod-de1-kopia-vdblz-maintain-job-1718592161gb4h2   0/1     Completed   0                10h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-1718583892q2ns   0/1     Completed   0                12h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-171858778rx9km   0/1     Completed   0                11h
postman-backup-prod-de1-kopia-fw9b4-maintain-job-171859346hd5r9   0/1     Completed   0                10h
repo-maintain-job-1718583275574-fjvdx                             0/1     Completed   0                13h
repo-maintain-job-1718583575574-9dxbj                             0/1     Completed   0                12h
repo-maintain-job-1718583587497-59dzn                             0/1     Completed   0                12h
repo-maintain-job-1718583595539-c49bk                             0/1     Completed   0                12h
repo-maintain-job-1718583875575-xsdq4                             0/1     Completed   0                12h
repo-maintain-job-1718583881845-9jwsm                             0/1     Completed   0                12h
repo-maintain-job-1718583886866-btw8n                             0/1     Completed   0                12h
repo-maintain-job-1718584475577-ms46g                             0/1     Completed   0                12h
repo-maintain-job-1718584488615-ktw7z                             0/1     Completed   0                12h
repo-maintain-job-1718584494633-5sjqv                             0/1     Completed   0                12h
repo-maintain-job-1718584513718-vkh4h                             0/1     Completed   0                12h
repo-maintain-job-1718584533746-6gq8g                             0/1     Completed   0                12h
repo-maintain-job-1718584540761-ccmv2                             0/1     Completed   0                12h
repo-maintain-job-1718586875584-g4d6q                             0/1     Completed   0                12h
repo-maintain-job-1718587175585-8jm8l                             0/1     Completed   0                11h
repo-maintain-job-1718587475585-tzsnf                             0/1     Completed   0                11h
repo-maintain-job-1718587483390-rvfpx                             0/1     Completed   0                11h
repo-maintain-job-1718587489403-bpfkg                             0/1     Completed   0                11h
repo-maintain-job-1718587496427-mk87t                             0/1     Completed   0                11h
repo-maintain-job-1718587775585-whwg8                             0/1     Completed   0                11h
repo-maintain-job-1718588075586-7l2l9                             0/1     Completed   0                11h
repo-maintain-job-1718588097309-kgjw2                             0/1     Completed   0                11h
repo-maintain-job-1718588375587-8ddvp                             0/1     Completed   0                11h
repo-maintain-job-1718588400297-bp8bb                             0/1     Completed   0                11h
repo-maintain-job-1718588415359-6g2sw                             0/1     Completed   0                11h
repo-maintain-job-1718588423385-lkddt                             0/1     Completed   0                11h
repo-maintain-job-1718590475592-9wdgv                             0/1     Completed   0                11h
repo-maintain-job-1718590775593-kls5g                             0/1     Completed   0                10h
repo-maintain-job-1718592174416-4298z                             0/1     Completed   0                10h
repo-maintain-job-1718593379857-kz96d                             0/1     Completed   0                10h
repo-maintain-job-1718593386910-6fsdg                             0/1     Completed   0                10h
repo-maintain-job-1718593396937-55vpc                             0/1     Completed   0                10h
repo-maintain-job-1718593435141-l4kmt                             0/1     Completed   0                10h
repo-maintain-job-1718593443182-wjv6t                             0/1     Completed   0                10h
repo-maintain-job-1718593475326-ftj4r                             0/1     Completed   0                10h
repo-maintain-job-1718593548635-w8m5t                             0/1     Completed   0                10h
repo-maintain-job-1718593573752-b9xb4                             0/1     Completed   0                10h
repo-maintain-job-1718593587806-nr2sc                             0/1     Completed   0                10h
repo-maintain-job-1718593596321-87w5k                             0/1     Completed   0                10h
search-backup-prod-de1-kopia-52bbc-maintain-job-1718585675pbgsm   0/1     Completed   0                12h
search-backup-prod-de1-kopia-52bbc-maintain-job-17185892756kpff   0/1     Completed   0                11h
search-backup-prod-de1-kopia-52bbc-maintain-job-1718593601bh4vp   0/1     Completed   0                10h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-171858625mdpz   0/1     Completed   0                12h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-17185898g2z2p   0/1     Completed   0                11h
temporal-backup-prod-de1-kopia-p72q7-maintain-job-17185935dsdnl   0/1     Completed   0                10h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171858452rqqhf   0/1     Completed   0                12h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171858840dmvvc   0/1     Completed   0                11h
theming-backup-prod-de1-kopia-28wkd-maintain-job-171859337m9jld   0/1     Completed   0                10h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-171858597558c469z   0/1     Completed   0                12h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-171858957558rtlkb   0/1     Completed   0                11h
wams-backup-prod-de1-kopia-jdj9c-maintain-job-1718593609917bkf7   0/1     Completed   0                10h

What did you expect to happen: There should be only 3 maintain-job pods per backup repositories

The following information will help us better understand what’s going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

  • kubectl logs deployment/velero -n velero
  • velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
  • velero backup logs <backupname>
  • velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
  • velero restore logs <restorename>

Anything else you would like to add:

velero-server logs -> https://gist.github.com/veerendra2/a25ba2e55070230af7cf791350f327f8

Environment: velero-server

          command:
            - /velero
          args:
            - server
            ### Flags
            - --uploader-type=kopia
            - --default-backup-storage-location=backup
            - --log-format=json
            - --keep-latest-maintenance-jobs=1
            ### Global Flags
            - --features=EnableCSI
            - --default-backup-ttl=${velero_schedule_ttl}
            - --resource-timeout=60m
            - --terminating-resource-timeout=60m

node-agent

      containers:
        - args:
            - node-agent
            - server
            - --data-mover-prepare-timeout=60m
            - --resource-timeout=60m
            - --features=EnableCSI
            - --log-format=json
          command:
            - /velero
  • Velero version (use velero version): Server v1.13.2
  • Velero features (use velero client config get features):
  • Kubernetes version (use kubectl version): Server Version: v1.28.5
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration: Azure AKS
  • OS (e.g. from /etc/os-release): Ubuntu

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

Please share the velero log bundle from the time when the maintenance jobs were created to the latest.