terraform-provider-azurerm: r/kubernetes_cluster: api returns InvalidIdentityValues during update

Quote from Azure support:

This is a known issue and we are working internally to solve it. In the meanwhile could you please try to use private flight using this link, once the portal opens try to change one of the setting on your aks cluster: https://portal.azure.com/?feature.canmodifystamps=true&Microsoft_Azure_ContainerService=flight1#home . The fix will be merged to prod soon.

The workaround works because it triggers the update logic for identity property. If terraform thinks the identity property is updated, it will generate a correct identity body. Any letter case change can trigger this updating logic. Related code part: https://github.com/terraform-providers/terraform-provider-azurerm/blob/v2.49.0/azurerm/internal/services/containers/kubernetes_cluster_resource.go#L1120
https://github.com/terraform-providers/terraform-provider-azurerm/blob/v2.49.0/azurerm/internal/services/containers/kubernetes_cluster_resource.go#L1835

This can be a temporary workaround before the issue is fully fixed.

Personally, I would like a fix in azure-sdk-for-go to permanently fix this. The reason for this issue is introduced in this comment: https://github.com/terraform-providers/terraform-provider-azurerm/issues/10406#issuecomment-791930148 . In AKS’s swagger, we have defined userAssignedIdentities.additionalProperties.properties.principalId and userAssignedIdentities.additionalProperties.properties.clientId as readonly. In the specification, readonly is defined as:

Relevant only for Schema “properties” definitions. Declares the property as “read only”. This means that it MAY be sent as part of a response but SHOULD NOT be sent as part of the request.

It should be SDK’s responsibility to handle this part of work. It’s quite common for developers to GET an object and then PUT it, it’s not reasonable to let SDK user to manually handle this. For some language, such as python, the auto-generated SDK already has the ability to set readonly properties to none before sending the request, but that’s not the case for go. I think some manual work need to be done in azure-sdk-for-go, the related issue: https://github.com/Azure/azure-sdk-for-go/issues/14478

We have exactly the same issue as described here and opened a ticket with MSFT (ref. 2104080050000277)

After some testing the MSFT engineer discovered that the resourceId of the identity contains a uppercase g in resourcegroups.

Working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity”

Non-working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity”

So if I use the following configuration (see below) it will actually deploy and update.

resource "azurerm_user_assigned_identity" "uai" {
  resource_group_name = "my_rg"
  location            = "westeurope"
  name                = "my_identity"
}

data "azurerm_subscription" "current" {}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "my_aks"
  location            = "westeurope"
  resource_group_name = "my_rg"

  identity {
    type                      = "UserAssigned"
    user_assigned_identity_id = "${data.azurerm_subscription.current.id}/resourcegroups/${azurerm_user_assigned_identity.uai.resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${azurerm_user_assigned_identity.uai.name}"
  }

In #10296 the resourceId has been changed to have the uppercase G… Azure isn’t really clear whether it should contain a g or G. Because if you look in the portal at the properties of the user assigned identity resource it will show you the resourceId with a G. But when you use powershell of az-cli to see the resourceId it will show you the lowercase g.

Please do note that I don’t consider this as a good workaround… But hopefully it shines a light on what is going wrong here.

Also, this worked for me without having to write out the whole id string.

user_assigned_identity_id = replace(azurerm_user_assigned_identity.r_aksUserIdentity.id,"resourceGroups","resourcegroups")

We have exactly the same issue as described here and opened a ticket with MSFT (ref. 2104080050000277) After some testing the MSFT engineer discovered that the resourceId of the identity contains a uppercase g in resourcegroups. Working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity” Non-working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity” So if I use the following configuration (see below) it will actually deploy and update.

resource "azurerm_user_assigned_identity" "uai" {
  resource_group_name = "my_rg"
  location            = "westeurope"
  name                = "my_identity"
}

data "azurerm_subscription" "current" {}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "my_aks"
  location            = "westeurope"
  resource_group_name = "my_rg"

  identity {
    type                      = "UserAssigned"
    user_assigned_identity_id = "${data.azurerm_subscription.current.id}/resourcegroups/${azurerm_user_assigned_identity.uai.resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${azurerm_user_assigned_identity.uai.name}"
  }

In #10296 the resourceId has been changed to have the uppercase G… Azure isn’t really clear whether it should contain a g or G. Because if you look in the portal at the properties of the user assigned identity resource it will show you the resourceId with a G. But when you use powershell of az-cli to see the resourceId it will show you the lowercase g. Please do note that I don’t consider this as a good workaround… But hopefully it shines a light on what is going wrong here.

Also, this worked for me without having to write out the whole id string.

user_assigned_identity_id = replace(azurerm_user_assigned_identity.r_aksUserIdentity.id,"resourceGroups","resourcegroups")

This worked for me as well. Thank you @BartVanBerkel and @aging-dh Cheers!

I can confirm that the workaround works, but I end up in the same issue as with #11223 that it wants to do an update each time I run terrafrom apply asit wants to change the user_assigned_identity_id from resourceGroups to resourcegroups.

We have exactly the same issue as described here and opened a ticket with MSFT (ref. 2104080050000277)

After some testing the MSFT engineer discovered that the resourceId of the identity contains a uppercase g in resourcegroups.

Working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity”

Non-working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity”

So if I use the following configuration (see below) it will actually deploy and update.

resource "azurerm_user_assigned_identity" "uai" {
  resource_group_name = "my_rg"
  location            = "westeurope"
  name                = "my_identity"
}

data "azurerm_subscription" "current" {}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "my_aks"
  location            = "westeurope"
  resource_group_name = "my_rg"

  identity {
    type                      = "UserAssigned"
    user_assigned_identity_id = "${data.azurerm_subscription.current.id}/resourcegroups/${azurerm_user_assigned_identity.uai.resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${azurerm_user_assigned_identity.uai.name}"
  }

In #10296 the resourceId has been changed to have the uppercase G… Azure isn’t really clear whether it should contain a g or G. Because if you look in the portal at the properties of the user assigned identity resource it will show you the resourceId with a G. But when you use powershell of az-cli to see the resourceId it will show you the lowercase g.

Please do note that I don’t consider this as a good workaround… But hopefully it shines a light on what is going wrong here.

We have exactly the same issue as described here and opened a ticket with MSFT (ref. 2104080050000277) After some testing the MSFT engineer discovered that the resourceId of the identity contains a uppercase g in resourcegroups. Working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity” Non-working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity” So if I use the following configuration (see below) it will actually deploy and update.

resource "azurerm_user_assigned_identity" "uai" {
  resource_group_name = "my_rg"
  location            = "westeurope"
  name                = "my_identity"
}

data "azurerm_subscription" "current" {}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "my_aks"
  location            = "westeurope"
  resource_group_name = "my_rg"

  identity {
    type                      = "UserAssigned"
    user_assigned_identity_id = "${data.azurerm_subscription.current.id}/resourcegroups/${azurerm_user_assigned_identity.uai.resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${azurerm_user_assigned_identity.uai.name}"
  }

In #10296 the resourceId has been changed to have the uppercase G… Azure isn’t really clear whether it should contain a g or G. Because if you look in the portal at the properties of the user assigned identity resource it will show you the resourceId with a G. But when you use powershell of az-cli to see the resourceId it will show you the lowercase g. Please do note that I don’t consider this as a good workaround… But hopefully it shines a light on what is going wrong here.

Also, this worked for me without having to write out the whole id string.

user_assigned_identity_id = replace(azurerm_user_assigned_identity.r_aksUserIdentity.id,"resourceGroups","resourcegroups")

Sorry for the thumbs down! I just wanted to flag that the work around is not working for me. I’m still getting the original The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property. error.

$ terraform0.14 version
Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/azurerm v2.56.0

I forgot to update… but you are 100% correct. In some update scenarios the ‘workaround’ doesn’t work. As said: “Please do note that I don’t consider this as a good workaround…”

We have exactly the same issue as described here and opened a ticket with MSFT (ref. 2104080050000277) After some testing the MSFT engineer discovered that the resourceId of the identity contains a uppercase g in resourcegroups. Working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity” Non-working MI ID: “/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my_rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my_identity” So if I use the following configuration (see below) it will actually deploy and update.

resource "azurerm_user_assigned_identity" "uai" {
  resource_group_name = "my_rg"
  location            = "westeurope"
  name                = "my_identity"
}

data "azurerm_subscription" "current" {}

resource "azurerm_kubernetes_cluster" "aks" {
  name                = "my_aks"
  location            = "westeurope"
  resource_group_name = "my_rg"

  identity {
    type                      = "UserAssigned"
    user_assigned_identity_id = "${data.azurerm_subscription.current.id}/resourcegroups/${azurerm_user_assigned_identity.uai.resource_group_name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${azurerm_user_assigned_identity.uai.name}"
  }

In #10296 the resourceId has been changed to have the uppercase G… Azure isn’t really clear whether it should contain a g or G. Because if you look in the portal at the properties of the user assigned identity resource it will show you the resourceId with a G. But when you use powershell of az-cli to see the resourceId it will show you the lowercase g. Please do note that I don’t consider this as a good workaround… But hopefully it shines a light on what is going wrong here.

Also, this worked for me without having to write out the whole id string.

user_assigned_identity_id = replace(azurerm_user_assigned_identity.r_aksUserIdentity.id,"resourceGroups","resourcegroups")

Sorry for the thumbs down! I just wanted to flag that the work around is not working for me. I’m still getting the original The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property. error.

$ terraform0.14 version
Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/azurerm v2.56.0

Is there a link to the Microsoft ticket so we can track it?