velero: [Kopia] Backup partially failing with timeout error after 24 hours despite default timeout of 4h

What steps did you take and what happened:

A scheduled backup is running. Backup get stuck for 24 hours and fails with timeout error. In backup description I see some pod volumes backup completed but many missing. They are not even listed in failed or new section of description.

Logs:

<details>
  <summary>
    Expand for logs
  </summary>
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otac-0
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="timed out waiting for all PodVolumeBackups to complete" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/podvolume/backupper.go:317" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:448" name=otac-0
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-2wbc8
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-2wbc8
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-2xjrt
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-2xjrt
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-4cmnl
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-4cmnl
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-7bt6z
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-7bt6z
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-7cks5
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-7cks5
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-7h77k
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-7h77k
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-9bxjs
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-9bxjs
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-9hs7f
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-9hs7f
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-d8244
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-d8244
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-dfhhm
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-dfhhm
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-f4wzm
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-f4wzm
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-fjp2m
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-fjp2m
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-fx6n2
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-fx6n2
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-gmr2p
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-gmr2p
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-hfpcr
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-hfpcr
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-hms76
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-hms76
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-hpxfl
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-hpxfl
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-kvhdm
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-kvhdm
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-l82dv
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-l82dv
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-mhlsb
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-mhlsb
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-mt42z
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-mt42z
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-nzp95
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-nzp95
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-qn47w
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-qn47w
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-qn9jw
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-qn9jw
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-stw2s
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-stw2s
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-svhx2
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-svhx2
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otawg-deployment-bf74b684f-t4m9m
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otawg-deployment-bf74b684f-t4m9m
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-admin-0
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-admin-0
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-0
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-0
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-1
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-1
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-10
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-10
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-11
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-11
time="2024-09-15T13:29:06Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-12
time="2024-09-15T13:29:06Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-12
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-13
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-13
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-14
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-14
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-15
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-15
time="2024-09-15T13:29:07Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-16
time="2024-09-15T13:29:07Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-16
time="2024-09-15T13:29:08Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-17
time="2024-09-15T13:29:08Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-17
time="2024-09-15T13:29:08Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-18
time="2024-09-15T13:29:08Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-18
time="2024-09-15T13:29:08Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-19
time="2024-09-15T13:29:08Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-19
time="2024-09-15T13:29:09Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-2
time="2024-09-15T13:29:09Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-2
time="2024-09-15T13:29:09Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-20
time="2024-09-15T13:29:09Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-20
time="2024-09-15T13:29:09Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-21
time="2024-09-15T13:29:09Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-21
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-22
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-22
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-23
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-23
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-24
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-24
time="2024-09-15T13:29:10Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-25
time="2024-09-15T13:29:10Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-25
time="2024-09-15T13:29:11Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-26
time="2024-09-15T13:29:11Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-26
time="2024-09-15T13:29:11Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-27
time="2024-09-15T13:29:11Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-27
time="2024-09-15T13:29:11Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-3
time="2024-09-15T13:29:11Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-3
time="2024-09-15T13:29:12Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-4
time="2024-09-15T13:29:12Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-4
time="2024-09-15T13:29:12Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-5
time="2024-09-15T13:29:12Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-5
time="2024-09-15T13:29:12Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-6
time="2024-09-15T13:29:12Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-6
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-7
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-7
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-8
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-8
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-backend-search-9
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-backend-search-9
time="2024-09-15T13:29:13Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-0
time="2024-09-15T13:29:13Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-0
time="2024-09-15T13:29:14Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-1
time="2024-09-15T13:29:14Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-1
time="2024-09-15T13:29:14Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-10
time="2024-09-15T13:29:14Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-10
time="2024-09-15T13:29:14Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-11
time="2024-09-15T13:29:14Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-11
time="2024-09-15T13:29:15Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-12
time="2024-09-15T13:29:15Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-12
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-13
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-13
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-14
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-14
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-15
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-15
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-16
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-16
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-17
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-17
time="2024-09-15T13:29:16Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-18
time="2024-09-15T13:29:16Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-18
time="2024-09-15T13:29:17Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-19
time="2024-09-15T13:29:17Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-19
time="2024-09-15T13:29:17Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-2
time="2024-09-15T13:29:17Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-2
time="2024-09-15T13:29:17Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-20
time="2024-09-15T13:29:17Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-20
time="2024-09-15T13:29:18Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-21
time="2024-09-15T13:29:18Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-21
time="2024-09-15T13:29:18Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-22
time="2024-09-15T13:29:18Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-22
time="2024-09-15T13:29:18Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-23
time="2024-09-15T13:29:18Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-23
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-24
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-24
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-25
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-25
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-26
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-26
time="2024-09-15T13:29:19Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-3
time="2024-09-15T13:29:19Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-3
time="2024-09-15T13:29:20Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-4
time="2024-09-15T13:29:20Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-4
time="2024-09-15T13:29:20Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-5
time="2024-09-15T13:29:20Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-5
time="2024-09-15T13:29:20Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-6
time="2024-09-15T13:29:20Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-6
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-7
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-7
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-8
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-8
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otcs-frontend-9
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otcs-frontend-9
time="2024-09-15T13:29:21Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-0
time="2024-09-15T13:29:21Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-0
time="2024-09-15T13:29:22Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-1
time="2024-09-15T13:29:22Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-1
time="2024-09-15T13:29:22Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-2
time="2024-09-15T13:29:22Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-2
time="2024-09-15T13:29:22Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=otiv-amqp-3
time="2024-09-15T13:29:22Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=otiv-amqp-3
time="2024-09-15T13:29:24Z" level=info msg="1 errors encountered backup up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 logSource="pkg/backup/backup.go:444" name=trident-controller-6b96f5dbcf-m4lqb
time="2024-09-15T13:29:24Z" level=error msg="Error backing up item" backup=velero/sch-backup-bag-prod-daily-20240914132659 error="failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/nodeagent/node_agent.go:62" error.function=github.com/vmware-tanzu/velero/pkg/nodeagent.IsRunningInNode logSource="pkg/backup/backup.go:448" name=trident-controller-6b96f5dbcf-m4lqb
</details>

What did you expect to happen:

Backup to work fine.

The following information will help us better understand what’s going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

  • kubectl logs deployment/velero -n velero
  • velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
  • velero backup logs <backupname>
  • velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
  • velero restore logs <restorename>

Anything else you would like to add:

Environment:

  • Velero version (use velero version): 1.12.3
  • Velero features (use velero client config get features): None
  • Kubernetes version (use kubectl version): 1.28
  • Kubernetes installer & version: GKE 1.28
  • Cloud provider or hardware configuration: GCP
  • OS (e.g. from /etc/os-release): Ubuntu 22.04 with containerd

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (7 by maintainers)

Most upvoted comments

In the v1.14 doc you can see following… you’d just configure for emptyDir and nothing else.

- conditions:
    volumeTypes:
      - emptyDir
      - downwardAPI
      - configmap
      - cinder
  action:
    type: skip