velero: Azure Government Cloud/backup-location: Confidential Client is not supported in Cross Cloud request

What steps did you take and what happened:

I followed the documentation at: https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure#setup

I used the Service Principal authentication method and installed velero client in a container of the AKS cluster.

I copied my kubeconfig file and created a credentials-velero file:

AZURE_SUBSCRIPTION_ID=****************************************
AZURE_TENANT_ID=****************************************
AZURE_CLIENT_ID=****************************************
AZURE_CLIENT_SECRET=****************************************
AZURE_RESOURCE_GROUP=My_Resource_Group
AZURE_CLOUD_NAME=AzureUSGovernmentCloud

I ran:

$ velero install \
     --provider azure \
     --plugins velero/velero-plugin-for-microsoft-azure:v1.2.0 \
     --bucket velero \
     --secret-file /root/credentials-velero \
     --backup-location-config resourceGroup=My_Resource_Group,storageAccount=My_Storage_Account,subscriptionId=**************************************** \
     --snapshot-location-config apiTimeout=300,resourceGroup=My_Resource_Group,subscriptionId=****************************************

CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: attempting to create resource client
W0511 18:50:16.667780     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client
W0511 18:50:16.698221     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client
W0511 18:50:16.713965     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client
W0511 18:50:16.771095     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client
W0511 18:50:16.787557     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client
W0511 18:50:16.805095     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource client
W0511 18:50:16.857974     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: attempting to create resource client
W0511 18:50:16.922281     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: attempting to create resource client
W0511 18:50:16.958081     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client
W0511 18:50:17.011261     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client
W0511 18:50:17.028847     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
W0511 18:50:17.036395     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
[...]
W0511 18:51:17.206350     498 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Namespace/velero: attempting to create resource
Namespace/velero: attempting to create resource client
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: attempting to create resource client
W0511 18:51:17.269503     498 warnings.go:67] rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: attempting to create resource client
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: attempting to create resource client
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: attempting to create resource client
BackupStorageLocation/default: created
VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: attempting to create resource client
VolumeSnapshotLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: attempting to create resource client
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

$ velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX   PHASE     LAST VALIDATED   ACCESS MODE   DEFAULT
default   azure      velero          Unknown   Unknown          ReadWrite     true

What did you expect to happen:

$ velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX   PHASE     LAST VALIDATED   ACCESS MODE   DEFAULT
default   azure      velero          Available   2021-05-11 XX:XX:XX          ReadWrite     true

The output of the following commands will help us better understand what’s going on:

• kubectl logs deployment/velero -n velero

time="2021-05-11T18:52:09Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:166"
time="2021-05-11T18:52:25Z" level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.usgovcloudapi.net/subscriptions/****************************************/resourceGroups/****************************************/providers/Microsoft.Storage/storageAccounts/****************************************/listKeys?%24expand=kerb&api-version=2019-06-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"AADSTS900382: Confidential Client is not supported in Cross Cloud request.\\r\\nTrace ID: fffea4cf-5c7d-48bc-be6e-e0f6c8dd0d01\\r\\nCorrelation ID: 4c789c15-3faa-4dcd-a906-d51e730feeb2\\r\\nTimestamp: 2021-05-11 18:52:25Z\",\"error_codes\":[900382],\"timestamp\":\"2021-05-11 18:52:25Z\",\"trace_id\":\"fffea4cf-5c7d-48bc-be6e-e0f6c8dd0d01\",\"correlation_id\":\"4c789c15-3faa-4dcd-a906-d51e730feeb2\"}" error.file="/go/src/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:217" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_sync_controller.go:175"
time="2021-05-11T18:52:50Z" level=error msg="Error getting a backup store" backup-storage-location=default controller=backup-storage-location error="rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.usgovcloudapi.net/subscriptions/****************************************/resourceGroups/****************************************/providers/Microsoft.Storage/storageAccounts/****************************************/listKeys?%24expand=kerb&api-version=2019-06-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {\"error\":\"invalid_request\",\"error_description\":\"AADSTS900382: Confidential Client is not supported in Cross Cloud request.\\r\\nTrace ID: 1fe3ed4d-d58b-4ab3-bd64-3638cc2d0900\\r\\nCorrelation ID: 95c6ed79-4b9d-4126-b69c-3169aac32bb9\\r\\nTimestamp: 2021-05-11 18:52:50Z\",\"error_codes\":[900382],\"timestamp\":\"2021-05-11 18:52:50Z\",\"trace_id\":\"1fe3ed4d-d58b-4ab3-bd64-3638cc2d0900\",\"correlation_id\":\"95c6ed79-4b9d-4126-b69c-3169aac32bb9\"}" error.file="/go/src/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:217" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_storage_location_controller.go:100"

Environment:

• Velero version (use velero version):

Client:
        Version: v1.6.0
        Git commit: 5bd70fd8eef316d220317245e46dc6016c348dce

• Velero features (use velero client config get features):

features: <NOT SET>

• Kubernetes version (use kubectl version):

On my laptop (not in the container):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1-5-g76a04fc", GitCommit:"2513fdbb36e2ddf13bc0b17460151c03eb3a3547", GitTreeState:"clean", BuildDate:"2021-04-09T04:34:48Z", GoVersion:"go1.15.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.9", GitCommit:"6c90dbd9d6bb1ae8a4c0b0778752be06873e7c55", GitTreeState:"clean", BuildDate:"2021-03-22T23:02:49Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:

AKS on the Texas US Gov cloud

• Cloud provider or hardware configuration:

Microsoft Azure Government

$ kubectl get nodes -o wide
NAME                                STATUS   ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
aks-agentpool-14984447-vmss000000   Ready    agent   7d    v1.19.9   10.0.12.4     <none>        Ubuntu 18.04.5 LTS   5.4.0-1043-azure   docker://19.3.14
aks-agentpool-14984447-vmss000001   Ready    agent   7d    v1.19.9   10.0.12.115   <none>        Ubuntu 18.04.5 LTS   5.4.0-1043-azure   docker://19.3.14
aks-agentpool-14984447-vmss000002   Ready    agent   7d    v1.19.9   10.0.12.226   <none>        Ubuntu 18.04.5 LTS   5.4.0-1043-azure   docker://19.3.14

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• Additional info

I was able to use the storage account with azcopy from the container using a shared access signature (SAS). az sync in both directions, az list and az remove works.

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

• 👍 for “I would like to see this bug fixed as soon as possible”
• 👎 for “There are more important bugs to focus on right now”