velero: velero backup logs hangs forever - More debug log needed in velero server.

What steps did you take and what happened: Trying to get the logs of a backup fail and the command hangs forever.

What did you expect to happen: To see the logs generated for that particular backup

The output of the following commands will help us better understand what’s going on:

• kubectl logs deployment/velero -n velero

time="2021-04-27T08:20:38Z" level=error msg="Error updating download request" controller=download-request downloadRequest=velero/velero-1619435026-dailybackups-20210426110358-20210426151139 error="downloadrequests.velero.io \"velero-1619435026-dailybackups-20210426110358-20210426151139\" not found" logSource="pkg/controller/download_request_controller.go:81"
time="2021-04-27T08:20:58Z" level=error msg="Error updating download request" controller=download-request downloadRequest=velero/velero-1619435026-dailybackups-20210426110358-20210426142201 error="downloadrequests.velero.io \"velero-1619435026-dailybackups-20210426110358-20210426142201\" not found" logSource="pkg/controller/download_request_controller.go:81"
time="2021-04-27T08:21:09Z" level=error msg="Error updating download request" controller=download-request downloadRequest=velero/velero-1619435026-dailybackups-20210426110358-20210426151211 error="downloadrequests.velero.io \"velero-1619435026-dailybackups-20210426110358-20210426151211\" not found" logSource="pkg/controller/download_request_controller.go:81"
time="2021-04-27T08:21:26Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114"
time="2021-04-27T08:21:26Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121"
time="2021-04-27T08:21:41Z" level=error msg="Error updating download request" controller=download-request downloadRequest=velero/velero-1619435026-dailybackups-20210426110358-20210426125922 error="downloadrequests.velero.io \"velero-1619435026-dailybackups-20210426110358-20210426125922\" not found" logSource="pkg/controller/download_request_controller.go:81"
time="2021-04-27T08:21:52Z" level=info msg="Processing new ServerStatusRequest" controller=server-status-request logSource="pkg/controller/server_status_request_controller.go:90" phase= serverStatusRequest=velero/velero-cli-mlqp9
time="2021-04-27T08:22:27Z" level=info msg="Validating backup storage location" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114"
time="2021-04-27T08:22:27Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121"

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml

Name:         velero-1619435026-dailybackups-20210426110358
Namespace:    velero
Labels:       app.kubernetes.io/instance=velero-1619435026
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=velero
              helm.sh/chart=velero-2.17.2
              velero.io/schedule-name=velero-1619435026-dailybackups
              velero.io/storage-location=default
Annotations:  helm.sh/hook=post-install,post-upgrade
              helm.sh/hook-delete-policy=before-hook-creation
              velero.io/source-cluster-k8s-gitversion=v1.18.2
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  backupData=daily

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2021-04-26 12:04:03 +0100 BST
Completed:  2021-04-26 12:04:19 +0100 BST

Expiration:  2021-05-26 12:04:01 +0100 BST

Total items to be backed up:  7
Items backed up:              7

Velero-Native Snapshots: 1 of 1 snapshots completed successfully (specify --details for more information)

• velero backup logs <backupname>
• Hangs

Velero version Client: Version: v1.6.0 Git commit: - Server: Version: v1.6.0

Anything else you would like to add: The issue seems to be the command DownloadRequest never receives a URL so keeps looping waiting for that field to not be empty. velero has been deployed to a private AKS cluster

Environment:

• Velero features (use velero client config get features): features: <NOT SET>
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:18:00Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version: Azure Private AKS
• Cloud provider or hardware configuration: Azure
• OS (e.g. from /etc/os-release): Tried on both Mac and Ubuntu 20.04

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

• 👍 for “I would like to see this bug fixed as soon as possible”
• 👎 for “There are more important bugs to focus on right now”