azure-cli: az aks create fails to obtain SP credentials

I still have the same error if I try to create a AKS cluster in the portal or just by running a command like: az aks create -n my-cluster -g MyResGroup

BUT… I can create the AKS cluster with the following steps:

1. create a service principal and get the informations az ad sp create-for-rbac --skip-assignment

It will show something like:

{ “appId”: “4567876c-47d4-438b-8341-67g1a3d12rae”, “displayName”: “azure-cli-2019-06-15-22-24-12”, “name”: “http://azure-cli-2019-06-15-22-24-12”, “password”: “xzzg445667-efef-78888-769e-d4543567”, “tenant”: “fb245196-9867-4a04-rtf-4556ad3ddff6” }

1. Autoscaler feature, only if you need the feature, aks-preview must be installed: az extension add --name aks-preview Help: https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler

2. create a resource group: az group create --name MyResourceGrp --location westeurope

3. create the AKS cluster:

az aks create –resource-group MyResourceGrp
–name MyClusterName -s Standard_B2s –node-count 1 –generate-ssh-keys –service-principal 4567876c-47d4-438b-8341-67g1a3d12rae –client-secret xzzg445667-efef-78888-769e-d4543567 –enable-vmss –enable-cluster-autoscaler –min-count 1 –max-count 2

Could we close this now that we have a doc reference for this issue? It’s a known issue captured here: https://github.com/Azure/AKS/issues/1206

Requires a fix from Azure Active Directory which should provided updates on the issue above.

I’ve run this command three times, and the first 2 it failed with the above error and the third time it looks like it’s working. So …

This is still occurring, unfortunately. Of course, manually creating an SP and providing it via az cli args does work, but this previously worked without the need to do so.

Multiple attempts at this also pollutes the AppID list found here: https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade

After cleaning these up, I made an attempt at scripting up the process based on the older Azure DevOps utils script (https://github.com/Azure/azure-devops-utils/blob/master/bash/create-service-principal.sh). NOTE: this is super deprecated, but had some good boilerplate code to reference.

Essentially the gist is to create an SP, but store the output as a JSON environment variable for the session in order to use it when running the cluster create process (and so we can clean up after ourselves later avoiding the problem above).

# Change this to the resource group name you want to use, or want to create
RESOURCE_GROUP=default 

# Change this to the region you'll be using if creating a resource group
REGION=eastus

# Note I'm naïvely grabbing the first sub (`[0]`, counting from zero)
# edit as needed if you've got more
SUBSCRIPTION=$(az account list | jq -r .[0].id)

# Create the resource group to house AKS, or skip this step if you've already got one
az group create -g $RESOURCE_GROUP -l $REGION --output none

# Storing the JSON to grab two vars next
SERVICE_PRINCIPAL=$(az ad sp create-for-rbac \
 --scope /subscriptions/$SUBSCRIPTION/resourceGroups/$RESOURCE_GROUP \
 --role Contributor \
 --output json) 

# Get the AKS SP ID from the service principal JSON
AKS_SP_ID=$(echo $SERVICE_PRINCIPAL | jq -r '.appId')

# Get the AKS SP pass from the service principal JSON
AKS_SP_PASS=$(echo $SERVICE_PRINCIPAL | jq -r '.password')

You can then use both vars to create the cluster without echoing anything to logs/screen etc if automating or testing.

Example:

CLUSTER_NAME=default # Change as required

az aks create \
 --resource-group $RESOURCE_GROUP \
 --name $CLUSTER_NAME \
 --kubernetes-version 1.14.8 \
 --node-vm-size Standard_D2_v2 \
 --node-count 1 \
 --service-principal $AKS_SP_ID \
 --client-secret $AKS_SP_PASS \
 --output none

There’s probably a more succinct way to accomplish this, but I was in a bit of a rush 😅

Hi. I have similar problem here on my mac os:

I created the SP manually and the PASSWORD is returning not as a GUID like, but in this format bellow:

{ “appId”: “(SOME APP ID)”, “displayName”: “odaibert”, “name”: “http://test”, “password”: “V3#!\8nL:,<"Gj<~Lew4GlB.Oq35GO%m”, “tenant”: “(SOME TENANT ID)” }

CLIENTSECRET=“V3#!\8nL:,<"Gj<~Lew4GlB.Oq35GO%m”

Then I tried to create an AKS cluster using:

$ az aks create -n $CLUSTERNAME -g $RGNAME --kubernetes-version 1.16.9 --service-principal $SPNAME --client-secret $CLIENTSECRET --generate-ssh-keys --location $LOCATION --node-count 3 --debug

And I’m getting this error:

msrest.exceptions : Operation failed with status: 'Bad Request'. Details: Service principal client secret has invalid characters: ’ cli.azure.cli.core.util : Operation failed with status: ‘Bad Request’. Details: Service principal client secret has invalid characters: ' Operation failed with status: 'Bad Request'. Details: Service principal client secret has invalid characters: ’ Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x10b356430>] `

Any ideas??

Hi All - the reported issue where an AKS create command fails due to a non existent SP (even though it was created) is actually a data replication issue. As noted, if you retry the command the command will work (this happens to the portal as well).

The cause is a data replication lag between the create call for the SP, confirmation, and then replication to the requested region. This replication can take up to 4 minutes. This is also why commands that passed previously fail.

We are bringing the replication timing issue to the attention of the AD team.

Hi. I have similar problem here on my mac os:

I created the SP manually and the PASSWORD is returning not as a GUID like, but in this format bellow:

{ “appId”: “(SOME APP ID)”, “displayName”: “odaibert”, “name”: “http://test”, “password”: "V3#!\8nL:,<“Gj<~Lew4GlB.Oq35GO%m”, “tenant”: “(SOME TENANT ID)” }

CLIENTSECRET="V3#!\8nL:,<“Gj<~Lew4GlB.Oq35GO%m”

Then I tried to create an AKS cluster using:

$ az aks create -n $CLUSTERNAME -g $RGNAME --kubernetes-version 1.16.9 --service-principal $SPNAME --client-secret $CLIENTSECRET --generate-ssh-keys --location $LOCATION --node-count 3 --debug

And I’m getting this error:

msrest.exceptions : Operation failed with status: 'Bad Request'. Details: Service principal client secret has invalid characters: ’ cli.azure.cli.core.util : Operation failed with status: ‘Bad Request’. Details: Service principal client secret has invalid characters: ' Operation failed with status: 'Bad Request'. Details: Service principal client secret has invalid characters: ’ Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x10b356430>] `

Any ideas??

I have the same issue now! I just install Azure CLI 2.6.0 and when I try manually create a service principal like this (from doc): az ad sp create-for-rbac --skip-assignment --name myAKSClusterServicePrincipal

Got: { “appId”: “appId”, “displayName”: “displayName”, “name”: “name”, “password”: “9bzD}N{K7j2Wchss,o+fKdk}n8QP+|r:”, “tenant”: “tenantGUID” }

Then I use password to specify a service principal for an AKS cluster: az aks create --resource-group “resource-group-name” --name “cluster-name” --service-principal “appId” --client-secret “9bzD}N{K7j2Wchss,o+fKdk}n8QP+|r:”

Got error: Operation failed with status: ‘Bad Request’. Details: The credentials in ServicePrincipalProfile were invalid. Please see https://aka.ms/aks-sp-help for more details. (Details: adal: Refresh request failed. Status Code = ‘401’)

OR

Operation failed with status: ‘Bad Request’. Details: Service principal client secret has invalid characters:

According to an Azure support engineer, another workaround is resetting the SP password manually via az ad sp credential reset --name NAME --password PASSWORD where PASSWORD has at least one special character but no single quote. Then the AKS creation should succeed.

It seems that an ARM deployment for Kubernetes will always fail with the following error when the service principal secret contains quote-like characters:

{'additionalProperties': {}, 'code': 'InvalidTemplateDeployment', 'message': "The template deployment 'kubernetes-cluster' is not valid according to the validation procedure. The tracking id is 'xxx'. See inner errors for details.", 'target': None, 'details': [{'additionalProperties': {}, 'code': 'BadRequest', 'message': 'Provisioning of resource(s) for container service XXX in resource group XXX fa
iled. Message: {                                                                                                                                                                                                                     
  "code": "BadRequest",                                                                                                                                                                                                              
  "message": "Service principal client secret has invalid characters: ` \'",                                                                                                                                                         
  "target": "servicePrincipalProfile.secret"                                                                                                                                                                                         
 }. Details: ', 'target': None, 'details': None, 'additionalInfo': None}], 'additionalInfo': None}

This happens with the CLI versions 2.5.1, 2.6.0, and 2.7.0. We’ve also tried URL escaping the secret, but then Azure just states that the secret is incorrect because it doesn’t match the actual password. For now we’ve solved this by simply regenerating the principal password until we had one without quotes.

The fix is just being merged, it will be included in the next Azure CLI release.

Same issue with

azure-cli                          2.6.0 *

command-modules-nspkg              2.0.3
core                               2.6.0 *
nspkg                              3.0.4
telemetry                          1.0.4

Python location '/usr/local/opt/python/bin/python3.7'
Extensions directory '/Users/mo/.azure/cliextensions'

Python (Darwin) 3.7.7 (default, Mar 10 2020, 15:43:33)
[Clang 11.0.0 (clang-1100.0.33.17)]

I keep getting this, so as mentioned above, instead of;

 az aks create -n tye --generate-ssh-keys --node-count 1 --node-vm-size Standard_B2s

I use this instead;

az ad sp create-for-rbac --skip-assignment -n mySP
az aks create -n tye --generate-ssh-keys --node-count 1 --node-vm-size Standard_B2s --service-principal <appId-from-previous-command> --client-secret <password-from-previous-command>

Just want to chime in and say this is still an issue. Tried it last week, and got the error. Rerunning the command worked.

@jnoller, I am from Azure Data team, we have a dependency on AKS, and some users experienced this issue, it would be nice if we could get an ETA for this so that we can decide how do we proceed. Thanks!

Seems there are two issues. This issue: #https://github.com/MicrosoftDocs/azure-docs/issues/32883 which was closed when referenced to here is actually the issue I experience, i.e setting up the aad integration for users in another tenant.

az aks create … –aad-client-app-id $aadClientAppId This is the client id not found, but definitely exists --aad-server-app-id $aadServerAppId –aad-server-app-secret $aadServerAppSecret --aad-tenant-id $env:aadTenantId

If not same cause should be a separate issue.

I am witnessing this exact same behaviour. First two attempts failed but simply retrying the same command worked on 3rd attempt. Using Azure Cloud Shell with Bash.

Confirmed working now! Time to retire the old script 😄

Command:

az aks create -n my-cluster -g default

Output (after about 5 or so minutes):

{
  "aadProfile": null,
  "addonProfiles": null,
  "agentPoolProfiles": [
    {
      "availabilityZones": null,
      "count": 3,
      "enableAutoScaling": null,
      "enableEncryptionAtHost": false,
      "enableFips": false,
      "enableNodePublicIp": false,
      [...]

Thanks for pinging @navba-MSFT!

@davis-x Apologies for the late reply. Could you use the most recent version of the CLI and let us know if you are still facing this issue ? Awaiting your reply.

Still happening. Found this via the AKS QuickStart Guide, which one would hope would Just Work…

Can you help me ?

Operation failed with status: ‘Bad Request’. Details: The credentials in ServicePrincipalProfile were invalid. Please see https://aka.ms/aks-sp-help for more details. (Details: adal: Refresh request failed. Status Code = ‘400’. Response body:

This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.

I’m triying to make this lab of course: 40565G : https://github.com/microsoft/MCW-Modernizing-Data-Analytics-with-SQL-Server-2019/blob/master/Hands-on lab/Before the HOL - Modernizing Data Analytics with SQL Server 2019.md

It seems some aks commands use the stored service principal by default. I could workaround by creating and specifying the SP in the option. https://github.com/Azure/azure-cli/issues/9585#issuecomment-502542000

For me, removing the ~/.azure/aksServicePrincipal.json file prior to deployment works very well.

• Patriek van Dorp

From: tanaka_733 notifications@github.com Sent: Tuesday, October 22, 2019 7:10:15 AM To: Azure/azure-cli azure-cli@noreply.github.com Cc: Patriek van Dorp Patriek.van.Dorp@microsoft.com; Comment comment@noreply.github.com Subject: Re: [Azure/azure-cli] az aks create fails to obtain SP credentials (#9585)

I also face this error when following this doc. https://docs.microsoft.com/en-us/azure/aks/windows-container-clihttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Fwindows-container-cli&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184440503&sdata=V3fOW%2FHcc4nOV%2F68KedJtLLgzkRoQs6DOoUW2EP%2BaXM%3D&reserved=0

This doc is not helpful because it doesn’t explain what I should do to fix the issue at all. https://docs.microsoft.com/en-us/azure/aks/troubleshooting#im-receiving-errors-that-my-service-principal-was-not-found-when-i-try-to-create-a-new-cluster-without-passing-in-an-existing-onehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Ftroubleshooting%23im-receiving-errors-that-my-service-principal-was-not-found-when-i-try-to-create-a-new-cluster-without-passing-in-an-existing-one&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184440503&sdata=Dqljum%2BcYuWLwVQGs0cEtNv9xSI3xoDcUJXKtSVefRA%3D&reserved=0

I hope the AKS team will fix the issue by changing the az CLI command.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-cli%2Fissues%2F9585%3Femail_source%3Dnotifications%26email_token%3DAALCHLT7Q77PS6CJD2LMTLTQP2DLPA5CNFSM4HVHO4BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB4RQWY%23issuecomment-544807003&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184450496&sdata=mvxc9pckOY5tw2kebWyPKfyBpa%2BguAYhcU2VZqpBUho%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAALCHLUTSVSJM5KP2XQS3RLQP2DLPANCNFSM4HVHO4BA&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184450496&sdata=QTcO0BL0mZIIUCEzwvRNKxjPPg2itrnIhOW%2F9Tm2Qak%3D&reserved=0.

experiencing same @GLink worked back in early may, and no more…

@marchanddesable as a temporary workaround I manually created a service principal then passed in the details into the az aks create command, which worked