terraform-provider-azurerm: Azure KeyVault Error: Provider produced inconsistent result after apply

We are facing the same issue when creating secrets via azurerm_key_vault_secret with terraform azurerm provider 3.5.0.

Terraform version v0.13.4 Azurerm provider 2.88.1 Following and hoping for a fix. We are experiencing the same issue with our pipeline sporadically. The keyvault gets created during an earlier stage in the pipeline and these secrets get created during a later stage in the pipeline. The secrets appear in the portal but not in state. Error: Provider produced inconsistent result after apply When applying changes to module.SQL.azurerm_key_vault_secret.sql_password, provider “registry.terraform.io/hashicorp/azurerm” produced an unexpected new value: Root resource was present, but now absent.

I’ve also noticed during times when this occurs, we are also not able to complete an az keyvault list either even though the keyvault is shown in the portal. Once cli can list keyvault again, we are able to re-run pipeline successfully.

Breaking discovery!!! If you add a tag to the keyvault, prior to running tf it will be able to successfully find the kv.

I had similar issues with the subscription scoped context. I’ve created a Key Vault provider that connects to the KV data plane directly, maybe some folks here find it useful.

https://registry.terraform.io/providers/tiwood/azurekeyvault/latest/docs

@magodo I just tested and sadly it doesn’t work for me with 3.12.0 and tf 1.2.4 or I do something wrong. ☹️ It’s stuck at searching for Resource Group.

Error: retrieving the Resource ID the Key Vault at URL "https://myvault.vault.azure.net/": retrieving Vault: 
(Name "myvault" / Resource Group "myvaultrg"): keyvault.VaultsClient#Get: Failure responding to request: 
StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound"
Message="Resource group 'myvaultrg' could not be found."

With specifying provider on data azurerm_key_vault_secret then it works, which makes me wonder if https://github.com/hashicorp/terraform-provider-azurerm/pull/17407 even could have solved that - it was on azurerm_key_vault, but problem is on _secret|_certificate|_key no?

provider "azurerm" {
  subscription_id = "sss_default"
  ... the usual stuff
}

provider "azurerm" {
  alias           = "myprovider"
  subscription_id = "sss_foreign"
  ... the usual stuff
}

data "azurerm_key_vault" "vault" {
  provider            = azurerm.myprovider
  name                = "myvault"
  resource_group_name = "myvaultrg"
}

data "azurerm_key_vault_secret" "secret" {
  provider     = azurerm.myprovider # doesn't work without this, error above
  name         = "mysecret"
  key_vault_id = data.azurerm_key_vault.vault.id
}

Note that resource azurerm_key_vault_secret fails as well - it actually creates the secret, but then fails to verify it’s presence, as reported in https://github.com/hashicorp/terraform-provider-azurerm/issues/11782:

When applying changes to azurerm_key_vault_secret.psk, provider
"registry.terraform.io/hashicorp/azurerm" produced an unexpected new value:
Root resource was present, but now absent.

Just to document another scenario where this problem appears to occur, similar to others here.

We have a central Key Vault in which we store shared secrets. We were using data sources to pull back account details that were then used to create TLS certs that are then loaded into a subscription specific key vault. In our case, we missed explicitly adding the provider to the data source (which still worked), but that somehow caused the wrong subscription ID to be used when refreshing the state on the Key Vault (or secrets within it).

Hi guys, I had exactly the same issue too, with azurerm provider 2.99.0 and 3.9.0.

I was passing ID of a keyvault directly as a string, then I wonder maybe I am passing wrong value and tried to read keyvault with data source and pass ID parameter from there. Then I realized I can’t read that keyvault as I am not on correct subscription…

So az account set -s ... or rather setting subscription with the provider did the trick for me.

Pretty misleading error message I would say 😃

@Nemeczek do you have Key vault on a different subscription where you are creating resource. In this call, use module and change provider when calling it

everything is in one sub. only in different resource groups

I had providers described like this:

provider "azurerm" {
  subscription_id = module.estate.subscription.id
  features {}
}

provider "azurerm" {
  features {}
  alias           = "hub"
  subscription_id = module.estate.hub_subscription.id
}

And was attempting to create a certificate like this, which worked for the create:

resource "azurerm_key_vault_certificate" "winrm" {
  ...
}

But failed on subsequent plan operations.

I was expecting it to use the unaliased provider with subscription_id sourced from, in my case, module.estate.subscription.id but it was actually trying with the subscription_id assigned to the hub aliased provider, as judged by TF_LOG = Debug, much to my surprise. This resulted in the ResourceGroupNotFound 404 error already reported.

I tried:

resource "azurerm_key_vault_certificate" "winrm" {
  provider = azurerm
  ...
}

But it still failed, so I added another provider as a repeat of my unaliased provider:

provider "azurerm" {
  subscription_id = module.estate.subscription.id
  alias           = "automations"
  features {}
}

And reconfigured the certificate:

resource "azurerm_key_vault_certificate" "winrm" {
  provider = azurerm.automations
  ...
}

And it worked, although now I have two providers for essentially the same ARM endpoint? I don’t necessarily want to alias my default local provider though as conceptually it relates to a known configuration tier, and it means more lines of boilerplate.

It’s surprisingly inconsistent, please fix 🙏🙏

@Tbohunek Indeed, whilst currently the provider aims to work in a subscription level. The client of each service is initialized with a fixed subscription (the same as the provider): https://github.com/hashicorp/terraform-provider-azurerm/blob/6257739685f7729e9af8ce5e059d74e2d06f3ad3/internal/services/keyvault/client/client.go#L23. Some of the reason is the limitation of the underlying Azure SDK used, where the methods didn’t expose the subscription id as a parameter. However, this might be changed after we switch to use https://github.com/hashicorp/go-azure-sdk.

@Tbohunek Yes, in cross subscription use, the “other resource types” works because their id has the subscription id in it. While as the id of the key vault nested items are using the data plane id, which doesn’t have the sub id. So when reading them, the provider assumes the subscription is the current provider’s subscription.

I’d like to confirm that the config in https://github.com/hashicorp/terraform-provider-azurerm/issues/11059#issuecomment-1172423670 works with specifying the provider alias.

We have the same problem, (a little bit different we provide the “azurerm provider” to a separate module). The module then should sync keys from a key vault in a different subscription compared to the one we are using. However, somehow terraform believes the secrets are not there anymore and proceeds to delete them from the state. It then tries to re-add them but they are still there and this gives an error. Maybe our problem is a little bit more complex since we are running the code in an Azure devops pipeline.

module "test" {
  resource-group-kv = "test1"
 key-vault-name = "test2"
  providers = {
    azurerm = azurerm.sub
  }
}


provider "azurerm" {
  features {
    key_vault {
      purge_soft_delete_on_destroy = true
    }
  }
  alias           = "sub"
  subscription_id = "xxxxx"
}

In the module

data "azurerm_key_vault" "azvault" {
  name                = var.key-vault-name
  resource_group_name = var.resource-group-kv
}

resource "azurerm_key_vault_secret" "secret-developer-write" {
  name         = "super-secret"
  value        =  "secret"
  key_vault_id = data.azurerm_key_vault.azvault.id
}

We get the error:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

In case this helps anyone else but I received this error (Root resource was present, but now absent. and the error Cannot import non-existent remote object when attempting to import) too but in my case it was my own silly mistake because in my code I had multiple providers defined for different subscriptions.

When adding the secret I was passing the key vault resource id via a datasource that used the right provider, however in the module that creates the secret I was not specifying the provider so it was defaulting to the subscription for the default provider - which in my case is not where the key vault resides for the secret I was adding - I only noticed it when enabling the trace logging to see the actual API request that was made.

Another update is that I can (intermittently) reproduce this with following steps:

1. Check out and build one of my personal branch for the provider: https://github.com/magodo/terraform-provider-azurerm/tree/endpoint (diff against the main branch). The only diff is that this branch added an environment variable: ARM_RESOURCE_MANAGER_ENDPOINT_PREFIX, which allows users to target to a (region) specific ARM endpoint
2. Use above provider to provision the example template, which the azurerm_key_vault_secret commented out, and ARM_RESOURCE_MANAGER_ENDPOINT_PREFIX set to eastus
3. Uncomment the azurerm_key_vault_secret part and run terraform apply again, but this time with ARM_RESOURCE_MANAGER_ENDPOINT_PREFIX set to soemthing different than eastus, e.g. norwayeast

I ran above steps for 3 times, and the issue occurs 2 times.

So another way to get it fixed is to ensure the regional ARM endpoint used to create the key vault nested resources is the same as the resource group, which contains the key vault (based on this comment). There is one issue #15632 that asks for adding the regional endpoint support for the provider, just FYI.

We’re seeing a similar issue - https://github.com/hashicorp/terraform-provider-azurerm/issues/14249

We’ve pinned the issue down to a call made by the TF provider to the Azure REST API. This one:

GET https://management.azure.com/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resources?$filter=resourceType eq 'Microsoft.KeyVault/vaults' and name eq 'this-is-my-kv'&$top=5&api-version=2020-06-01

Depending on where the Terraform script is run from (i.e. the IP of the machine that the TF script is run on), it either works or doesn’t work.

Example:

• If the runner and Key Vault locations are the same (e.g. eastus), it works.
• If the runner and Key Vault locations are different (e.g. eastus/uksouth), it doesn’t work.

Another thing to mention is that when using the Azure REST API to directly fetch the Key Vault by ID, it works.

It’s purely this resource list filter call that has the issue.

Hope that helps resolve the issue.

@magodo I have run through the test as done previously, and I can confirm that this appears to have solved the issue for us! 👍🏽

@tplive and @aniro, is there any chance that the module used to create the KeyVault is not the in the same subscription as the one used to create the secret (just as @Aldiuser mentioned)?

@magodo both steps are configured in a similar way and use the same subscription in our case.

@tplive and @aniro, is there any chance that the module used to create the KeyVault is not the in the same subscription as the one used to create the secret (just as @Aldiuser mentioned)?

@magodo In our case, the the Vault has been previously created and exists in the same state, all in the same subscription.

@tplive Please send the config and trace log to sinbai@hotmail.com, thank you.