terraform-provider-azurerm: azurerm_key_vault not setting purge protection and error when attempting to purge keys.

The 409 Conflict is of interest, @jwshive is that consistent? We explicitly check the item is deleted before attempting to purge it, so this may indicate a bug or inconsistency between the read and purge operations.

@jackofallops

For us this is very consistent since this morning at least. Create environment, then destroy and we get this error. If we simply run destroy again we get a permission error (presumably the policy access was deleted successfully) and running a 3rd and all consecutive times reports the key vault cannot be found. Rerun create again and then destroy and the 409 error comes back.

So to summarise, the first destroy we run always gets a 409 on the key vault secrets now. We are running azurerm 2.47.0 and terraform 0.14.7. But I bumped the versions hoping it would fix it. This morning we were on azurerm 2.44.0 and terraform 0.14.4 with the exact same problem.

@mattduguid Yeah, i tested it with both and the behaviour was still funy. I did some tests just now. And terraform is deploying the KeyVault the right way. The problem was basically that attaching a KeyVault to an Azure Machine Learning workspace, automatically activates Purge Protection on that Keyvault. So Terraform deploys without Purge, then AzureML turns on the purge and then terraform tries to turn purge protection from true to nil, and it can’t do that because purge protection can’t be deactivated.

Then when destroying the environment, if you have the option “purge deleted keyvaults” set to true. Then AzureML (and maybe other attached services have this behaviour as well) turns on purge protection, terraform tries to purge on delete, and everything fails.

I’m trying to figure out why AzureML is activating purge protection on the attached key vault.

My previous problem seems to be resolved after upgrading to terraform 0.14.7 and azurerm 2.48.0, but I’m experiencing the 409 error now as well:

Error: purging Secret "admin-password" (Key Vault "https://...-kv.vault.azure.net/"): keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=409 Code="Conflict" Message="Secret is currently being deleted." InnerError={"code":"ObjectIsBeingDeleted"}

My provider settings are the following:

key_vault {
      recover_soft_deleted_key_vaults = false
      purge_soft_delete_on_destroy    = true
}

Setting purge_soft_delete_on_destroy to false solves my issue but this doesn’t solve my issue. It seems @jackofallops is on the right track with the comment about a potential bug or inconsistency between the read and purge operations

I am getting the same error on azurerm 2.43.0. I’m using a module to deploy virtual machines in Azure using GitLab CI and a service principal. I used to store the storage account access key in the module, but I’ve removed that feature. When I try to run the module again it attempts to delete the secret in the key vault (not the key vault itself), I get the following:

Error: purging Secret “sa-access-key” (Key Vault “https://…vault.azure.net/”): keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=403 – Original Error: autorest/azure: Service returned an error. Status=403 Code=“Forbidden” Message=“The user, group or application ‘appid=xxx;oid=xxx;iss=https://sts.windows.net/xxx/’ does not have secrets purge permission on key vault ‘…;location=westeurope’. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287” InnerError={“code”:“ForbiddenByPolicy”}

This is even though I am certain the service principal has the right access to the key vault:

I’m using the following config:

resource "azurerm_key_vault" "vre-kv" {
  name                = "${local.rootname}-kv"
  location            = var.location
  resource_group_name = azurerm_resource_group.vre-rg.name
  tenant_id           = data.azurerm_client_config.current.tenant_id

  enabled_for_disk_encryption     = true
  enabled_for_template_deployment = true
  enabled_for_deployment          = true
  enable_rbac_authorization       = false

  soft_delete_retention_days = 7
  purge_protection_enabled   = false
  sku_name                   = "standard"

  network_acls {
    default_action = "Allow"
    bypass         = "AzureServices"
  }
  tags = local.tags
}

resource "azurerm_key_vault_access_policy" "vre-kv-ap" {
  key_vault_id       = azurerm_key_vault.vre-kv.id
  tenant_id          = data.azurerm_client_config.current.tenant_id
  object_id          = data.azurerm_client_config.current.object_id
  key_permissions    = ["create", "get", "list", "delete", "purge", ]
  secret_permissions = ["set", "get", "delete", "list", "purge", ]

  depends_on = [
    azurerm_key_vault.vre-kv
  ]
}

And for the provider configuration:

provider "azurerm" {
  subscription_id = "xxx"

  features {
    virtual_machine {
      delete_os_disk_on_deletion = true
    }
    key_vault {
      recover_soft_deleted_key_vaults = false
      purge_soft_delete_on_destroy    = true
    }
  }
}

Our pipelines are dead in the water due to this bug and our heavy reliance on azure key vault. 😦

Using hashicorp/azurerm v2.41.0 I am also facing the same issue when destroying secrets for the first time. Getting the below error: keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=400 – Original Error: autorest/azure: Service returned an error. Status=400 Code=“NotSupported” Message=“Operation "purge" is not enabled for this vault.” This deletes secrets but does not delete KeyVault.

When running terraform destroy again after the first attempt, it destroys the key vault, but gives the following error: Message=“The user, group or application does not have secrets delete permission on key vault.” This deletes the resources but gives the error message.

I gave my service principal the purge permission as documented here and I still get the error posted in the opening notes.

My key vault sp permissions creation code is now

resource "azurerm_key_vault_access_policy" "sp-key_vault_access_policy" {
  key_vault_id = azurerm_key_vault.key_vault.id
  object_id = data.azurerm_client_config.current_client_config.object_id
  tenant_id = data.azurerm_client_config.current_client_config.tenant_id
  key_permissions = ["get", "list", "create", "encrypt", "delete", "purge"]
  secret_permissions = ["get", "list", "set", "delete", "purge"]
  depends_on = [azurerm_key_vault.key_vault]
}

Also, soft delete is enabled.

    "createMode": null,
    "enablePurgeProtection": null,
    "enableRbacAuthorization": false,
    "enableSoftDelete": true,
    "enabledForDeployment": false,
    "enabledForDiskEncryption": false,
    "enabledForTemplateDeployment": false,
    "networkAcls": null,
    "privateEndpointConnections": null,
    "provisioningState": "Succeeded",

Edit: When I posted the above I had forgot to enable soft delete. I enabled that in the key vault and when running terraform destroy I now see…

Error: purging Secret "sql-1-password" (Key Vault "https://my-key-vault.vault.azure.net/"): keyvault.BaseClient#PurgeDeletedSecret: Failure responding to request: StatusCode=409 -- 
Original Error: autorest/azure: Service returned an error. Status=409 Code="Conflict" Message="Secret is currently being deleted." InnerError={"code":"ObjectIsBeingDeleted"}

Which seems like it would be obvious during a destroy that the secret is being deleted.

same problem: using 2.54

The KV was created with purge_protection_enabled = false/null (or omit it), then on another run. It says purge protection is enabled AND like @daguito81 notes, this KV is being attached to a ML workspace.

Error: updating Vault: (Name "my-kv" / Resource Group "my-rg"): once Purge Protection has been Enabled it's not possible to disable it

terraform {
  required_version = "= 0.14.9"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 2.54"
    }
  }
}


data "azurerm_client_config" "current" {}


resource "azurerm_key_vault" "kv" {
  name                = "${var.name_prefix}-kv"
  location            = var.location
  resource_group_name = var.resource_group_name
  tenant_id           = data.azurerm_client_config.current.tenant_id

  sku_name                  = "standard"
  enable_rbac_authorization = true

  enabled_for_disk_encryption     = false
  enabled_for_template_deployment = false
  enabled_for_deployment          = false
  
  purge_protection_enabled = false
  soft_delete_retention_days = 30


  network_acls {
    bypass                     = "AzureServices"
    default_action             = "Deny"
    ip_rules                   = var.network_acl_ip_rules
    virtual_network_subnet_ids = var.network_acl_virtual_network_subnet_ids
  }

  tags = var.tags

}

Same for me on an apply: Error: purging Certificate “91-cert” (Key Vault “https://91-uks.vault.azure.net/”): keyvault.BaseClient#PurgeDeletedCertificate: Failure responding to request: StatusCode=400 – Original Error: autorest/azure: Service returned an error. Status=400 Code=“NotSupported” Message=“Operation "purge" is not enabled for this vault.”

Using azurem 2.41.0 and not setting purge_protection_enabled (so should have default of false).

Cert was removed though in above run (not unexpected due to code change) and subsequent attempt re-created it.