terraform-provider-azurerm: azurerm_databricks_workspace no longer working. storage identity and managed disk identity output values are blank

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave “+1” or “me too” comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.0.7

AzureRM Provider Version

3.45.0 and 3.52.0

Affected Resource(s)/Data Source(s)

azurerm_databricks_workspace

Terraform Configuration Files

just a week ago, this code worked perfectly. i used arm template examples to help guide deployment in terraform: https://github.com/Azure/azure-quickstart-templates/tree/master/quickstarts/microsoft.databricks


resource "azurerm_databricks_workspace" "databricks_workspace" {
  count = var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count

  location            = local.region[var.location]
  name                = "${var.group_prefix}${format("%02d", count.index)}-dbrickws"
  resource_group_name = var.resource_group_name
  tags                = local.tags

  customer_managed_key_enabled                        = var.cmk_enabled
  infrastructure_encryption_enabled                   = var.infrastructure_encryption_enabled
  managed_resource_group_name                         = "MW_${var.group_prefix}${format("%02d", count.index)}-dbrickws"
  managed_services_cmk_key_vault_key_id               = var.cmk_enabled ? azurerm_key_vault_key.databricks_services_key[count.index].id : null
  managed_disk_cmk_key_vault_key_id                   = var.cmk_enabled ? azurerm_key_vault_key.databricks_disks_key[count.index].id : null
  managed_disk_cmk_rotation_to_latest_version_enabled = var.cmk_enabled ? true : null
  network_security_group_rules_required               = var.nsg_rules_required
  public_network_access_enabled                       = var.public_network_access_enabled
  sku                                                 = var.sku

  custom_parameters {
    no_public_ip                                         = var.no_public_ip
    private_subnet_name                                  = replace(var.private_subnet_id, "/.*//", "")
    private_subnet_network_security_group_association_id = var.private_nsg_assoc_id
    public_subnet_name                                   = replace(var.public_subnet_id, "/.*//", "")
    public_subnet_network_security_group_association_id  = var.public_nsg_assoc_id
    storage_account_name                                 = "${replace(var.group_prefix, "-", "")}${format("%02d", count.index)}dbrickwssa"
    storage_account_sku_name                             = var.storage_account_sku_name
    virtual_network_id                                   = replace(var.private_subnet_id, "//subnets/.*/", "")
  }

  depends_on = [azurerm_key_vault_access_policy.databricks_service_policy]
}

resource "azurerm_databricks_workspace_customer_managed_key" "databricks_workspace_key" {
  count = var.cmk_enabled ? (var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count) : 0

  key_vault_key_id = azurerm_key_vault_key.databricks_dbfs_key[count.index].id
  workspace_id     = azurerm_databricks_workspace.databricks_workspace[count.index].id

  depends_on = [azurerm_key_vault_access_policy.databricks_policy]
}

//KV stuff

data "azurerm_client_config" "current" {}
data "azuread_service_principal" "databricks_spn" {
  display_name = "AzureDatabricks"
}

resource "azurerm_key_vault_key" "databricks_dbfs_key" {
  count = var.cmk_enabled ? (var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count) : 0
  name  = "${var.group_prefix}${format("%02d", count.index)}-dbricks-dbfs-key"

  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
  key_size     = 2048
  key_type     = "RSA"
  key_vault_id = var.keyvault_id
}

resource "azurerm_key_vault_key" "databricks_disks_key" {
  count = var.cmk_enabled ? (var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count) : 0
  name  = "${var.group_prefix}${format("%02d", count.index)}-dbricks-disks-key"

  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
  key_size     = 2048
  key_type     = "RSA"
  key_vault_id = var.keyvault_id
}

resource "azurerm_key_vault_key" "databricks_services_key" {
  count = var.cmk_enabled ? (var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count) : 0
  name  = "${var.group_prefix}${format("%02d", count.index)}-dbricks-services-key"

  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey",
  ]
  key_size     = 2048
  key_type     = "RSA"
  key_vault_id = var.keyvault_id
}

resource "azurerm_key_vault_access_policy" "databricks_policy" {
  count = var.cmk_enabled ? (var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count) : 0

  key_vault_id = var.keyvault_id
  object_id    = azurerm_databricks_workspace.databricks_workspace[count.index].storage_account_identity[0].principal_id
  tenant_id    = data.azurerm_client_config.current.tenant_id

  key_permissions = [
    "Get",
    "UnwrapKey",
    "WrapKey"
  ]
}

resource "azurerm_key_vault_access_policy" "databricks_service_policy" {
  count = var.cmk_enabled ? (var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count) : 0

  key_vault_id = var.keyvault_id
  object_id    = data.azuread_service_principal.databricks_spn.object_id
  tenant_id    = data.azurerm_client_config.current.tenant_id

  key_permissions = [
    "Get",
    "UnwrapKey",
    "WrapKey"
  ]
}

resource "azurerm_key_vault_access_policy" "databricks_disk_policy" {
  count = var.cmk_enabled ? (var.is_web_auth_workspace && var.resource_count > 0 ? 1 : var.resource_count) : 0

  key_vault_id = var.keyvault_id
  object_id    = azurerm_databricks_workspace.databricks_workspace[count.index].managed_disk_identity[0].principal_id
  tenant_id    = data.azurerm_client_config.current.tenant_id

  key_permissions = [
    "Get",
    "UnwrapKey",
    "WrapKey"
  ]
}

Debug Output/Panic Output

Error: Invalid index
on .terraform/modules/databricks/kv.tf line 61, in resource "azurerm_key_vault_access_policy" "databricks_policy":
  object_id    = azurerm_databricks_workspace.databricks_workspace[count.index].storage_account_identity[0].principal_id

Error: Missing required argument
with module.databricks.azurerm_key_vault_access_policy.databricks_disk_policy[0]
on .terraform/modules/databricks/kv.tf line 89, in resource "azurerm_key_vault_access_policy" "databricks_disk_policy":
  object_id    = azurerm_databricks_workspace.databricks_workspace[count.index].managed_disk_identity[0].principal_id
The argument "object_id" is required, but no definition was found.

Expected Behaviour

everything build fine a week ago

Actual Behaviour

the databricks resource no longer seems to be outputting storage and managed disk identities. i also see this error in the activity log:

\"code\": \"InvalidParameter\",\r\n \"message\": \"https://KEYVAULT.vault.azure.net//keys/td04ybf51t300-dbricks-disks-key/d8d33b7dff184990923b3b7207ed5625 is not a valid versioned Key Vault Key URL. It should be in the format https://<vaultEndpoint>/keys/<keyName>/<keyVersion>.\",\r\n \"target\": \"diskEncryptionSet.properties.activeKey.keyUrl\"\r\n }\r\n}"}]}]}}",

Steps to Reproduce

No response

Important Factoids

No response

References

No response