velero: Backups Not Being Deleted on Request or TTL

What steps did you take and what happened: Backup delete requested

What did you expect to happen: The backup was removed and not in Deleting

Debug bundle attached

Anything else you would like to add:

time="2023-10-09T06:32:50Z" level=info msg="deletion request 'dre-services-beta-daily-fs-20230930130357-7rpjx' removed." backup=dre-services-beta-daily-fs-20230930130357 controller=backup-deletion deletebackuprequest=velero/dre-services-beta-daily-fs-20230930130357-lrsx6 logSource="pkg/controller/backup_deletion_controller.go:500"
time="2023-10-09T06:32:50Z" level=error msg="Unable to download tarball for backup dre-services-beta-daily-fs-20230930130357, skipping associated DeleteItemAction plugins" backup=dre-services-beta-daily-fs-20230930130357 controller=backup-deletion deletebackuprequest=velero/dre-services-beta-daily-fs-20230930130357-lrsx6 error="error copying Backup to temp file: rpc error: code = Unknown desc = error getting object dre-services-beta/backups/dre-services-beta-daily-fs-20230930130357/dre-services-beta-daily-fs-20230930130357.tar.gz: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 210000+3559+93026522+4972629, host id: 210000+3559+93026522+4972629" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:774" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile logSource="pkg/controller/backup_deletion_controller.go:269"
time="2023-10-09T06:32:50Z" level=info msg="Removing PV snapshots" backup=dre-services-beta-daily-fs-20230930130357 controller=backup-deletion deletebackuprequest=velero/dre-services-beta-daily-fs-20230930130357-lrsx6 logSource="pkg/controller/backup_deletion_controller.go:293"
time="2023-10-09T06:32:50Z" level=info msg="Removing pod volume snapshots" backup=dre-services-beta-daily-fs-20230930130357 controller=backup-deletion deletebackuprequest=velero/dre-services-beta-daily-fs-20230930130357-lrsx6 logSource="pkg/controller/backup_deletion_controller.go:318"
time="2023-10-09T06:32:50Z" level=info msg="Founding existing repo" backupLocation=default logSource="pkg/repository/ensurer.go:85" repositoryType=kopia volumeNamespace=depsdb
time="2023-10-09T06:32:52Z" level=warning msg="active indexes [xs0_2efea80920463c895fa90bd121105d00-s779091e80502d4c8-c1 xs1_de2ccbfe718a2945e4c374f7b92de12c-s8bd40c898fc40152-c1 xs2_b95b4938b89bd10d7cc647da11ee661f-sdbcbee9ec3e3956d-c1 xs3_24b8597af0200c1e5b58abcfd7be9252-scb86b8dc77fa6d53-c1 xs4_e376fa4bce775cc4838537acbf9e2bd2-sfc9e83b9a6194d36-c1 xs5_1ec5e20307cc57e015c73dd5e4611d0b-s98fd747af26ad087-c1 xs6_2060d9294b4f2b84adec833831797bf9-s56a25c6f3e4024b7-c1 xn7_07ea36f814a51d4eb8422b7e6e929f43-s5cb433dc3b0da0e3121-c1 xn7_0839862359eb42a7b5420c7333d36e5b-s960fceba5b004fbd121-c1 xn7_0cf773c3d5fab29240288312a6d19c53-se2b7ef5bf2331ddf121-c1 xn7_0fc66b7646d3cb654fcde1150f5e2dbc-s5c838959d945e011121-c1 xn7_1345a35c8cc24e422d7d39967ce8b2b3-sa7bf147a9b61aaa6121-c1 xn7_15159143c974da15df512e45204f1af5-s6a4a64528a9b7094121-c1 xn7_23ba1624d9313fd56d1daac07288d1ae-s89fc20bd89b54972121-c1 xn7_2c9ad2d43c9378a9b904e9edd441b3b5-sfd492350541aab7a121-c1 xn7_32f4348d3db2f4fb38b42be7b47e3612-sf23532d3e758494b121-c1 xn7_332cf0048869618d3dd7b643e4c6b4ec-s4048eee862123186121-c1 xn7_35e7908adf15a2c78a1324672976eb44-sa31390de0b5f3744121-c1 xn7_3c1ee8c85d65bcd3a30ce17cfc56d08d-sae502d084b9c0786121-c1 xn7_3e1fa5371283c2ccec4bf667c21f4c3f-s84ac8ab41cb34e2e121-c1 xn7_4641a4401813b12fa7e288193896e33c-s51d98f40805d2ec8121-c1 xn7_49559032edb5d392c64b63b9fdd89739-s836f35fa8cd56abf121-c1 xn7_49783f9ebbac655872099d586475cc99-sf25de4627baf9b48121-c1 xn7_4a6b97315c288bf64032f322019e7718-s5815a301a50aa65c121-c1 xn7_535518d5a224818d5955fd54a748c715-s80ef2c8168294850121-c1 xn7_5364d8f5f7f64a9e1ff13d36f15da839-s0bd39be509c1615d121-c1 xn7_5adf67ebce787020f52f0af44f636ffe-sed3be410014d2430121-c1 xn7_5d125cadfda8abe21ce250773ec485ec-s090461d1647133dc121-c1 xn7_63c9f4fe05685cf38778ad844f4d912c-sf9b3abd5e815762e121-c1 xn7_65249b127a35a4d4c9261167a21ee971-se2eff919f451355b121-c1 xn7_65740d240776b884b9f399e45dead980-s3bc9c3755f7f78c4121-c1 xn7_6c25fd8d6ef52b00a832950a6c486020-s4cd5d910bf7445ee121-c1 xn7_6c554247acdf0673e5e3437454fb2a97-s5203b4e849764896121-c1 xn7_6e22ea92ee81e49473c6db6adcf3243d-se6bcd8bee6d20f02121-c1 xn7_6e2de2fc9ac9988ce42b81704df8c1a5-s05c27d27d2d31cfe121-c1 xn7_7e32598c88d998ad6bdd0a677c142292-sb4d0d8d8f5ebff34121-c1 xn7_8001f16fb2fa844cbee48ce8df0597ae-s158d9eb83c81b71e121-c1 xn7_84aae8cd118a2df5a30a1ec411fba91b-s96c07ca6bdb0603d121-c1 xn7_8abc9cb976de677d69f4db046ba85d07-sa38fb7da20b22dc5121-c1 xn7_8c5e390a84c80dade1772e816e89caa3-s290fc0172085f92f121-c1 xn7_9332efc0d36ae9aa415ae0b342e86c77-s722b2203d833c9b3121-c1 xn7_9435d46c244dd2f09994947a8fb7e21b-s3cfa67ea3f48f3e3121-c1 xn7_9c528199650b15d72df7544fa4351897-s66ad2b74e957de5f121-c1 xn7_af914859d2c51c5091e3c4b55d60c22e-sa3855dc62dbb188f121-c1 xn7_b71215d1d1cc3b6da7866a07ec6f9cd4-sd4728898ac4d0196121-c1 xn7_c856dbdc3abd4791f3733ed83c516cac-s4f688a11c3bd2801121-c1 xn7_cb39d19817c50ab973d5705b576eadf6-s17312c66a59a451f121-c1 xn7_cf6f024d031aae6e35cf5e6ff326debe-sa8241b6556338254121-c1 xn7_d5488229f3a7e0a5891e234acecd707a-scb522254da72d7f8121-c1 xn7_dbe091ed5a450108b443d938da7312ee-s84b6683d1e2e0613121-c1 xn7_e157c8ee3e229d5039ea4de6105e48db-s89debd8508a72960121-c1 xn7_e158d7ab417daae87c2f4b73765ec477-s020f01580794647a121-c1 xn7_e2b758a57aa1e8a6fc05726d7bdf40e7-s22187138af630e24121-c1 xn7_ea64ffedc1d475d4415949cf791d0383-s39b0228ab8572761121-c1 xn7_f44368c6aaa8efc665a6a427efa69e4d-s0f76f5e244421a38121-c1 xn7_f6fb14a1c70e2291be792ce5c024f90b-s0cd67fd6285a69da121-c1 xn7_f9541d2920a89e0618ef81a61d905ebc-s03ad704fd66e383f121-c1 xn7_fc35108f2e3b1d25b8a3b531212d824d-s8643151ed310a4eb121-c1 xn7_ff5c59f6c98d953315d239dc45cd5eec-s02c202021222e27b121-c1 xn8_0916b082aee15faec070c919e5062b3d-s3893245394335a9d121-c1 xn8_154aa2bf50f89eadc7f755a34e7b089c-se21ecccf31b6ebf0121-c1 xn8_25f649fb1ed4387990b5cbdd3badffb5-sbfb71613a87d9aa1121-c1 xn8_2da504c5c7c86ee8bcfbd10e6d1029c0-s8d5ecad22bb2f432121-c1 xn8_368db89d8c7c5b7808f8cd7a28f628ff-s840a24cdccc70f7e121-c1 xn8_37178c84d502c29171aaee3ad993d8dd-s888f883d4c860096121-c1 xn8_47f2dc97858b940de34a3ee819afb658-sefa5237d14b8445d121-c1 xn8_4cca9a040e25125399b604dca901d62f-s48393345217d1da6121-c1 xn8_6347b1d7808b1c29be9f09f6c8d21988-sfc4518962ca9c050121-c1 xn8_6b29a6b8a62a2145d20f64a832f44afb-s792b2dfb7c82a51c121-c1 xn8_7db0b3dd00dd52635521b5687f8a380d-s5264f3a35ebd0618121-c1 xn8_84030e5127ec336316891c3ce3cf745f-sddc8c5dba1ed2182121-c1 xn8_8ed7994518670cbc035f2ea76d76ee29-s24c939e78e9b48b4121-c1 xn8_9bb55a996c04071540f636c450dee8d2-sfa47b7380812741f121-c1 xn8_a11c860b46e8c6991698da67b0564ea8-s982e483ae483809e121-c1 xn8_a510c708fbe3e5beaf2e32ac0992adba-sbd75c842b9709e38121-c1 xn8_a7f1fea760c2c513f2e49597562a8147-s73427edbf4491efa121-c1 xn8_b9fbabb1880fe99b9c8d2a94512cd550-s5e53e9ddfa809ef8121-c1 xn8_bc8e71562de8b637a18da7a626b20cab-s6e54d3d1f73c29ba121-c1 xn8_cb46917ceafbf68116c5aaf77bbfc528-s1077b0c56503439f121-c1 xn8_d120d33b1255412cbd478f274aa1904e-s95f585dc01bc77fe121-c1 xn8_d242e111aa49338468daa8e7784c68c8-s029e6bf52d07564c121-c1] deletion watermark 2023-10-07 16:14:18 +0000 UTC" logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:101" sublevel=error

Environment:

  • Velero version (use velero version): 1.12.0
  • Velero features (use velero client config get features):
  • Kubernetes version (use kubectl version): 1.22
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration: NKE
  • OS (e.g. from /etc/os-release):

bundle-2023-10-09-07-43-06.tar.gz

Vote on this issue!

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 8
  • Comments: 22 (5 by maintainers)

Commits related to this issue

Most upvoted comments

@pseymournutanix These logs are expected.

The fix won’t work on the backups already in “Deleting” state, if only works for the new created backup deletion request.

I also have same issue with v1.12.2. Also affects backups without any errors.

FYI, I also have the same problems with v1.12.1

I’m having similar problems.

Seems to affect only backups that had errors - those without errors can be deleted (or expire) without problems.

Taking a quick look at the provided bundle, I noticed that the log content was not properly captured in the log file. Can you help to regenrate a new one that includes a full log for further troubleshooting of this issue.?

Based on current bundle, I found several error messages below recorded in describe file:

Errors:
    failed to wait BackupRepository: timed out waiting for the condition
    failed to wait BackupRepository: timed out waiting for the condition
    failed to wait BackupRepository: timed out waiting for the condition

Upon further examination of these errors, it apprears that Velero kept waiting for backup repository to become ready before hitting timeout.