external-dns: [azure] Pod fails to obtain token, when 2 MSI are assigned to VM scale set

What happened: Pod fails to refresh token when more than 1 user assigned Managed Identity is associated with the Virtual Machine Scale Set.

time=“2020-04-28T21:47:44Z” level=error msg=“azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/6100196f-6f28-4a23-9ab0-2e6ccc491527/resourceGroups/EastUS-EQS-AKS-SYSENG/providers/Microsoft.Network/dnsZones?api-version=2018-05-01: StatusCode=400 – Original Error: adal: Refresh request failed. Status Code = ‘400’. Response body: {"error":"invalid_request","error_description":"Identity not found"}”

What you expected to happen: Use the correct identity or at least show a message that the correct identity to use must be defined.

How to reproduce it (as minimally and precisely as possible): Start an external-dns pod in a cluster with the Virtual Machine Scale Set using 2 User assigned identities

Anything else we need to know?: Removing any of the assigned identities make it work, preferably remove the one that doesn’t have permissions to the DNS zone you want to manipulate.

azure.json: { “tenantId”: “cb333d92-redacted-bd”, “subscriptionId”: “6100196f-redacted-27”, “resourceGroup”: “EastUS-EQS-AKS-SYSENG”, “useManagedIdentityExtension”: true }

Adding to the azure.json “userAssignedIdentityID”: “404a9933-redacted-582”, didn’t work either (but I’m not sure what this option is used for)

Environment:

External-DNS version (use external-dns --version): v20200401-v0.7.1
DNS provider: Azure
Others:

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 11
Comments: 24 (5 by maintainers)

Most upvoted comments

As explained above, this issue occurred when more than 1 user assigned Managed Identity is associated with the Virtual Machine Scale Set. Adding a particular user assigned managed identity in the azure.json which is then used to create the secret resolved the issue for me.

{ “tenantId”: “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”, “subscriptionId”: “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”, “resourceGroup”: “resource-group-name”, “useManagedIdentityExtension”: true, “userAssignedIdentityID”: “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx” }

digitamind on Jul 19, 2020

@DigitaMind I was following current azure guide and I am not sure how to get the userAssignedIdentityID. Where do we get this from?

darkn3rd on Feb 23, 2021

@darkn3rd if you have Azure AADPodIdentity setup in the cluster then you can use the following configuration to bind external-dns to the managed identity:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: my-external-dns
  namespace: kube-system
spec:
  type: 0
  resourceID: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/my-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my-external-dns

  # az ad sp list --filter "displayName eq 'my-external-dns'" -o tsv --query '[].appId'
  clientID: 00000000-0000-0000-0000-000000000000
---
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: my-external-dns-binding
  namespace: kube-system
spec:
  azureIdentity: my-external-dns
  selector: my-external-dns

And for the external-dns pod, add the following label: aadpodidbinding: my-external-dns.

etiennetremel on Jun 25, 2021