pulumi: Azure secrets provider does not authenticate using Client credentials

I’ve been setting up pulumi to use azure keyvault.

Prior to this I had my CI (on azure devops) build something like this:

- script: |
    export PATH=$PATH:$HOME/.pulumi/bin
    pulumi login --cloud-url azblob://pulumi-state
    pulumi stack select test
    pulumi preview --diff
  displayName: run pulumi
  workingDirectory: src
  failOnStderr: true
  continueOnError: false
  env:
    ARM_CLIENT_ID: $(client_id)
    ARM_CLIENT_SECRET: $(client_secret)
    ARM_TENANT_ID: $(tenant_id)
    ARM_SUBSCRIPTION_ID: $(subscriptionId)
    AZURE_STORAGE_ACCOUNT: $(storage_account)
    AZURE_STORAGE_KEY: $(storage_account_key)
    PULUMI_CONFIG_PASSPHRASE: $(pulumi_config_passphrase)

I decided to move away from using passphrase, and use --secrets-provider. Created a stack fine, using the secrets provider, used my own user on azure to do that (just local commands).

However, now when I was trying to select the stack on the CI just using service principal authentication, pipeline is not able to authenticate against keyvault.

The error I’m getting is this error: getting secrets manager: secrets (code=InvalidArgument): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://somekeyvault.vault.azure.net/keys/keyname/someid/decrypt?api-version=7.0: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"} That’s without changing the above pipeline.

I’ve resolved the issue by authenticating with az cli as well as just passing sp credentials to pulumi.

- script: |
    az login --service-principal -u $(client_id) -p $(client_secret) --tenant $(tenant_id)
    export PATH=$PATH:$HOME/.pulumi/bin
    pulumi login --cloud-url azblob://pulumi-state
    pulumi stack select test
    pulumi preview --diff
  displayName: run pulumi
  workingDirectory: src
  failOnStderr: true
  continueOnError: false
  env:
    ARM_CLIENT_ID: $(client_id)
    ARM_CLIENT_SECRET: $(client_secret)
    ARM_TENANT_ID: $(tenant_id)
    ARM_SUBSCRIPTION_ID: $(subscriptionId)
    AZURE_STORAGE_ACCOUNT: $(storage_account)
    AZURE_STORAGE_KEY: $(storage_account_key)
    AZURE_KEYVAULT_AUTH_VIA_CLI: true

The documentation specifies that it should use Azure Environment authentication, which in my case should just use Client credentials just like what pulumi is using, however that is not the case.

About this issue

Original URL
State: open
Created 4 years ago
Comments: 16 (3 by maintainers)

Most upvoted comments

@Figglewatts - Thanks for this bit. I encountered the same problem and your post saved me a ton of troubleshooting 😃

I added the 3 AZURE variables in addition to the existing ARM ones.

nickpainter on Apr 17, 2020

Stumbled upon this while Googling for a separate issue - but have you tried using:

AZURE_CLIENT_ID: $(client_id)
AZURE_CLIENT_SECRET: $(client_secret)
AZURE_TENANT_ID: $(tenant_id)

I.e. replace ARM_ with AZURE_. See this link for more details.

figglewatts on Apr 9, 2020

I struggled with the same problem and finally I noticed the problem:

AZURE_KEYVAULT_AUTH_VIA_CLI: true

should be:

AZURE_KEYVAULT_AUTH_VIA_CLI: "true"

jancespivo on May 19, 2021

@fraPace As far as I’m aware, the Terraform provider uses the Azure REST API to interact with Azure (so not go-cloud), and Pulumi uses go-cloud. I totally agree with you that it’s a bit cumbersome to have to supply the same credentials twice.

As Pulumi simply calls into go-cloud without any specific behaviour for different kinds of secret managers (https://github.com/pulumi/pulumi/blob/v2.4.0/pkg/secrets/cloud/manager.go#L71), it would probably be better to fix this behaviour with a PR in go-cloud allowing it to also use ARM_ environment variables for authentication to key vault.

EDIT: In fact, I believe the best place to submit a PR to fix the behaviour would be: https://github.com/Azure/go-autorest/tree/master/autorest/azure/auth, as that’s the part that deals directly with the environment variables.

figglewatts on Jun 18, 2020

@Figglewatts I might be wrong here, but is Terraform using the go-cloud libraries or Pulumi? From the code that I scanned it looked like Pulumi. So to me it looks like Pulumi is using both and the login systems are different.

Should this be left to the end user? Not sure I fully agree with you. I do agree on the fact that the end users should provide the credentials in different ways (cli, env, etc). Is providing the same credentials twice becoming the new standard of software development?

fraPace on Jun 18, 2020

Hello @WackPenguin,

Thank you for the reply. This is what I just did from my own environment (it is no different from the pipeline environment since I use an self-hosted agent with the same image). I have run the following:

export ARM_CLIENT_ID="******"
export ARM_CLIENT_SECRET="****"

export ARM_SUBSCRIPTION_ID="****"
export ARM_TENANT_ID="******"

export AZURE_STORAGE_ACCOUNT="*****"
export AZURE_STORAGE_KEY="****"

export AZURE_KEYVAULT_AUTH_VIA_CLI=true

pulumi login -c azblob://*****
pulumi stack select dev
pulumi preview

I get the following error: error: getting secrets manager: open keeper azurekeyvault://*****: failed to Dial default KeyVault: Invoking Azure CLI failed with the following error: Please run 'az login' to setup account.

So my next actions is to add the following (according to the documentation of the Go secretManager that Pulumi is using):

export AZURE_TENANT_ID="****"
export AZURE_CLIENT_ID="*****"
export AZURE_CLIENT_SECRET="****"

The pulumi preview commands still return the same error: error: getting secrets manager: open keeper azurekeyvault://***: failed to Dial default KeyVault: Invoking Azure CLI failed with the following error: Please run 'az login' to setup account.

Therefore my last action is:

export AZURE_KEYVAULT_AUTH_VIA_CLI=false

In this case pulumi preview is working!

However, I would like to reiterate on the fact that it should not up to the final user to set up environment variables with redundant informations just because Pulumi is using Terraform on one side and some Go libraries on the other. The Pulumi Azure task is correctly set the ARM_ variables taking them from the Service Connector. However, Pulumi should standardize the way login is performed to use the different functionalities that it provides. So a quick fix should be to get the SP credentials and set (and unset) the environment variables.

@Figglewatts Thank you. I think for the moment we will do the same, but as I already stated above, I do not think it is the right approach. “Fixing” this issue should be pretty straightforward. Sadly I do not know Go very well, otherwise I would do a PR.

fraPace on Jun 18, 2020