velero: Error loading environment from AZURE_CREDENTIALS_FILE

What steps did you take and what happened: I have followed steps from this page: Part #3. Azure Kubernetes Services (AKS). Backup/Restore your AKS data with Velero.

What did you expect to happen: I was expecting that Velero will be installed to my Azure AKS cluster.

The output of the following commands will help us better understand what’s going on:

  • kubectl logs deployment/velero -n velero
time="2020-02-13T16:12:24Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:171"
time="2020-02-13T16:12:24Z" level=info msg="Starting Velero server v1.2.0 (5d008491bbf681658d3e372da1a9d3a21ca4c03c)" logSource="pkg/cmd/server/server.go:173"
time="2020-02-13T16:12:24Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:177"
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-microsoft-azure kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2020-02-13T16:12:25Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-microsoft-azure kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2020-02-13T16:12:25Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:337" namespace=velero
time="2020-02-13T16:12:25Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:343" namespace=velero
time="2020-02-13T16:12:27Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:372"
time="2020-02-13T16:12:27Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:406"
time="2020-02-13T16:12:27Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:413"
An error occurred: some backup storage locations are invalid: error getting backup store for location "default": rpc error: code = Unknown desc = error loading environment from AZURE_CREDENTIALS_FILE (/credentials/cloud): Can't separate key from value

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

  • Velero version (use velero version): Client: Version: v1.2.0 Git commit: 5d008491bbf681658d3e372da1a9d3a21ca4c03c
  • Velero features (use velero client config get features): features: <NOT SET>
  • Kubernetes version (use kubectl version): Client Version: version.Info{Major:“1”, Minor:“16”, GitVersion:“v1.16.1”, GitCommit:“d647ddbd755faf07169599a625faf302ffc34458”, GitTreeState:“clean”, BuildDate:“2019-10-02T17:01:15Z”, GoVersion:“go1.12.10”, Compiler:“gc”, Platform:“windows/amd64”} Server Version: version.Info{Major:“1”, Minor:“14”, GitVersion:“v1.14.5”, GitCommit:“0e9fcb426b100a2aea5ed5c25b3d8cfbb01a8acf”, GitTreeState:“clean”, BuildDate:“2019-08-05T09:13:08Z”, GoVersion:“go1.12.5”, Compiler:“gc”, Platform:“linux/amd64”}
  • Kubernetes installer & version: 1.14.5
  • Cloud provider or hardware configuration: Azure
  • OS (e.g. from /etc/os-release): Linux (ubuntu 16.04)

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 40 (21 by maintainers)

Most upvoted comments

Glad you got it working and thanks for the patience. I’m going to try to clarify the docs some more.

+3
skriss on May 5, 2020

I just want to be super clear that in the BSL, spec.config.storageAccountKeyEnvVar should have the value AZURE_STORAGE_ACCOUNT_ACCESS_KEY, not the actual access key. It stores the name of the env var, not the access key. Can you check that?

I did try this out and it now works for me, so I’m guessing that this is the final thing blocking you.

+3
skriss on May 5, 2020

Voila! That was it. Thanks, @skriss

+2
k8s42 on May 5, 2020

We got the fix merged already, so you could try getting an updated master image (note: you may need to change your image pull policy to Always for that initContainer to get the newest one).

+2
skriss on May 4, 2020

OK - it looks like you’re still missing one key part of configuring this, which is that in the BackupStorageLocation’s config, you need to add a key named storageAccountKeyEnvVar, whose value in this case would be AZURE_STORAGE_ACCOUNT_ACCESS_KEY. This is documented in some more detail here.

I think you can do this via the Helm chart by adding one more flag:

--set configuration.backupStorageLocation.config.storageAccountKeyEnvVar='AZURE_STORAGE_ACCOUNT_ACCESS_KEY'

If not, you can always kubectl -n velero edit backupstoragelocation azure and add storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY under spec.config.

+2
skriss on May 4, 2020

I, too, got it working with the help of this thread, thanks. The doumentation is a tad bit confusing in regards to passing one env var as reference to the var and the other by name; especially when the error message suggests that you passed the wrong value in storageAccountKeyEnvVar: config has invalid keys [storageAccountKeyEnvVar]; valid keys are [resourceGroup storageAccount subscriptionId blockSizeInBytes bucket prefix]

+1
michael-niemand on May 15, 2020

Yep, and that’s actually a bug.

I’ll get a PR up later today to fix.

+1
skriss on May 4, 2020

this feature hasn’t been released in a tagged version yet - if you want to use it, you’ll need to change your azure plugin image tag to master.

+1
skriss on May 4, 2020
Read more comments on GitHub
← velero: Error in creating CRD on openshift 3.11
velero: Error: unable to convert unstructured item to a v1 CRD: cannot convert int64 to float64 →