azure-cli: az aks create fails to obtain SP credentials
Describe the bug
Failed to create aks cluster using command line az aks create -n my-cluster -g test
Instead the cli fails to pull the service principal credentials
Operation failed with status: 'Bad Request'. Details: The credentials in ServicePrincipalProfile were invalid. Please see https://aka.ms/aks-sp-help for more details. (Details: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'f4b3caa9-defb-4ada-b190-e8422327afbb' was not found in the directory '599a411f-b08b-45fe-8545-623369f42d16'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: bb906908-bf98-46ad-ad5f-6262bd779100\r\nCorrelation ID: b5bb293a-d4be-43be-9a86-180b51515b4b\r\nTimestamp: 2019-06-06 18:45:03Z","error_codes":[700016],"timestamp":"2019-06-06 18:45:03Z","trace_id":"bb906908-bf98-46ad-ad5f-6262bd779100","correlation_id":"b5bb293a-d4be-43be-9a86-180b51515b4b","error_uri":"https://login.microsoftonline.com/error?code=700016"})
To Reproduce
Run az aks create -n my-cluster -g test
Expected behavior A cluster is created
Environment summary
Linux-4.9.87-linuxkit-aufs-x86_64-with-debian-buster-sid
Python 3.6.5
Shell: bash
azure-cli 2.0.66
Additional context https://docs.microsoft.com/en-us/azure/aks/kubernetes-service-principal#automatically-create-and-use-a-service-principal
The documentation clearly states that a principal should be created when one is not passed in.
In this case, a SP is being created, and I can see that after in the App Registrations panel, but yet the command is still failing, leaving me with a SP that appears to be useless
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 10
- Comments: 52 (10 by maintainers)
I still have the same error if I try to create a AKS cluster in the portal or just by running a command like: az aks create -n my-cluster -g MyResGroup
BUT… I can create the AKS cluster with the following steps:
It will show something like:
{ “appId”: “4567876c-47d4-438b-8341-67g1a3d12rae”, “displayName”: “azure-cli-2019-06-15-22-24-12”, “name”: “http://azure-cli-2019-06-15-22-24-12”, “password”: “xzzg445667-efef-78888-769e-d4543567”, “tenant”: “fb245196-9867-4a04-rtf-4556ad3ddff6” }
Autoscaler feature, only if you need the feature, aks-preview must be installed: az extension add --name aks-preview Help: https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler
create a resource group: az group create --name MyResourceGrp --location westeurope
create the AKS cluster:
az aks create –resource-group MyResourceGrp
–name MyClusterName -s Standard_B2s –node-count 1 –generate-ssh-keys –service-principal 4567876c-47d4-438b-8341-67g1a3d12rae –client-secret xzzg445667-efef-78888-769e-d4543567 –enable-vmss –enable-cluster-autoscaler –min-count 1 –max-count 2
Could we close this now that we have a doc reference for this issue? It’s a known issue captured here: https://github.com/Azure/AKS/issues/1206
Requires a fix from Azure Active Directory which should provided updates on the issue above.
I’ve run this command three times, and the first 2 it failed with the above error and the third time it looks like it’s working. So …
This is still occurring, unfortunately. Of course, manually creating an SP and providing it via az cli args does work, but this previously worked without the need to do so.
Multiple attempts at this also pollutes the AppID list found here: https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade
After cleaning these up, I made an attempt at scripting up the process based on the older Azure DevOps utils script (https://github.com/Azure/azure-devops-utils/blob/master/bash/create-service-principal.sh). NOTE: this is super deprecated, but had some good boilerplate code to reference.
Essentially the gist is to create an SP, but store the output as a JSON environment variable for the session in order to use it when running the cluster create process (and so we can clean up after ourselves later avoiding the problem above).
You can then use both vars to create the cluster without echoing anything to logs/screen etc if automating or testing.
Example:
There’s probably a more succinct way to accomplish this, but I was in a bit of a rush 😅
Hi. I have similar problem here on my mac os:
I created the SP manually and the PASSWORD is returning not as a GUID like, but in this format bellow:
{ “appId”: “(SOME APP ID)”, “displayName”: “odaibert”, “name”: “http://test”, “password”: “V3#!\8nL:,<"Gj<~Lew4GlB.Oq35GO%m”, “tenant”: “(SOME TENANT ID)” }
CLIENTSECRET=“V3#!\8nL:,<"Gj<~Lew4GlB.Oq35GO%m”
Then I tried to create an AKS cluster using:
$ az aks create -n $CLUSTERNAME -g $RGNAME --kubernetes-version 1.16.9 --service-principal $SPNAME --client-secret $CLIENTSECRET --generate-ssh-keys --location $LOCATION --node-count 3 --debug
And I’m getting this error:
msrest.exceptions : Operation failed with status: 'Bad Request'. Details: Service principal client secret has invalid characters:
’ cli.azure.cli.core.util : Operation failed with status: ‘Bad Request’. Details: Service principal client secret has invalid characters:' Operation failed with status: 'Bad Request'. Details: Service principal client secret has invalid characters:
’ Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x10b356430>] `Any ideas??
Hi All - the reported issue where an AKS create command fails due to a non existent SP (even though it was created) is actually a data replication issue. As noted, if you retry the command the command will work (this happens to the portal as well).
The cause is a data replication lag between the create call for the SP, confirmation, and then replication to the requested region. This replication can take up to 4 minutes. This is also why commands that passed previously fail.
We are bringing the replication timing issue to the attention of the AD team.
I have the same issue now! I just install Azure CLI 2.6.0 and when I try manually create a service principal like this (from doc): az ad sp create-for-rbac --skip-assignment --name myAKSClusterServicePrincipal
Got: { “appId”: “appId”, “displayName”: “displayName”, “name”: “name”, “password”: “9bzD}N{K7j2Wchss,o+fKdk}n8QP+|r:”, “tenant”: “tenantGUID” }
Then I use password to specify a service principal for an AKS cluster: az aks create --resource-group “resource-group-name” --name “cluster-name” --service-principal “appId” --client-secret “9bzD}N{K7j2Wchss,o+fKdk}n8QP+|r:”
Got error: Operation failed with status: ‘Bad Request’. Details: The credentials in ServicePrincipalProfile were invalid. Please see https://aka.ms/aks-sp-help for more details. (Details: adal: Refresh request failed. Status Code = ‘401’)
OR
Operation failed with status: ‘Bad Request’. Details: Service principal client secret has invalid characters:
According to an Azure support engineer, another workaround is resetting the SP password manually via
az ad sp credential reset --name NAME --password PASSWORD
wherePASSWORD
has at least one special character but no single quote. Then the AKS creation should succeed.It seems that an ARM deployment for Kubernetes will always fail with the following error when the service principal secret contains quote-like characters:
This happens with the CLI versions 2.5.1, 2.6.0, and 2.7.0. We’ve also tried URL escaping the secret, but then Azure just states that the secret is incorrect because it doesn’t match the actual password. For now we’ve solved this by simply regenerating the principal password until we had one without quotes.
The fix is just being merged, it will be included in the next Azure CLI release.
Same issue with
I keep getting this, so as mentioned above, instead of;
I use this instead;
Just want to chime in and say this is still an issue. Tried it last week, and got the error. Rerunning the command worked.
@jnoller, I am from Azure Data team, we have a dependency on AKS, and some users experienced this issue, it would be nice if we could get an ETA for this so that we can decide how do we proceed. Thanks!
Seems there are two issues. This issue: #https://github.com/MicrosoftDocs/azure-docs/issues/32883 which was closed when referenced to here is actually the issue I experience, i.e setting up the aad integration for users in another tenant.
az aks create … –aad-client-app-id $aadClientAppId
This is the client id not found, but definitely exists --aad-server-app-id $aadServerAppId
–aad-server-app-secret $aadServerAppSecret--aad-tenant-id $env:aadTenantId
If not same cause should be a separate issue.
I am witnessing this exact same behaviour. First two attempts failed but simply retrying the same command worked on 3rd attempt. Using Azure Cloud Shell with Bash.
Confirmed working now! Time to retire the old script 😄
Command:
az aks create -n my-cluster -g default
Output (after about 5 or so minutes):
Thanks for pinging @navba-MSFT!
@davis-x Apologies for the late reply. Could you use the most recent version of the CLI and let us know if you are still facing this issue ? Awaiting your reply.
Still happening. Found this via the AKS QuickStart Guide, which one would hope would Just Work…
Can you help me ?
Operation failed with status: ‘Bad Request’. Details: The credentials in ServicePrincipalProfile were invalid. Please see https://aka.ms/aks-sp-help for more details. (Details: adal: Refresh request failed. Status Code = ‘400’. Response body:
This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.
I’m triying to make this lab of course: 40565G : https://github.com/microsoft/MCW-Modernizing-Data-Analytics-with-SQL-Server-2019/blob/master/Hands-on lab/Before the HOL - Modernizing Data Analytics with SQL Server 2019.md
It seems some aks commands use the stored service principal by default. I could workaround by creating and specifying the SP in the option. https://github.com/Azure/azure-cli/issues/9585#issuecomment-502542000
For me, removing the ~/.azure/aksServicePrincipal.json file prior to deployment works very well.
From: tanaka_733 notifications@github.com Sent: Tuesday, October 22, 2019 7:10:15 AM To: Azure/azure-cli azure-cli@noreply.github.com Cc: Patriek van Dorp Patriek.van.Dorp@microsoft.com; Comment comment@noreply.github.com Subject: Re: [Azure/azure-cli] az aks create fails to obtain SP credentials (#9585)
I also face this error when following this doc. https://docs.microsoft.com/en-us/azure/aks/windows-container-clihttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Fwindows-container-cli&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184440503&sdata=V3fOW%2FHcc4nOV%2F68KedJtLLgzkRoQs6DOoUW2EP%2BaXM%3D&reserved=0
This doc is not helpful because it doesn’t explain what I should do to fix the issue at all. https://docs.microsoft.com/en-us/azure/aks/troubleshooting#im-receiving-errors-that-my-service-principal-was-not-found-when-i-try-to-create-a-new-cluster-without-passing-in-an-existing-onehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Faks%2Ftroubleshooting%23im-receiving-errors-that-my-service-principal-was-not-found-when-i-try-to-create-a-new-cluster-without-passing-in-an-existing-one&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184440503&sdata=Dqljum%2BcYuWLwVQGs0cEtNv9xSI3xoDcUJXKtSVefRA%3D&reserved=0
I hope the AKS team will fix the issue by changing the az CLI command.
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-cli%2Fissues%2F9585%3Femail_source%3Dnotifications%26email_token%3DAALCHLT7Q77PS6CJD2LMTLTQP2DLPA5CNFSM4HVHO4BKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB4RQWY%23issuecomment-544807003&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184450496&sdata=mvxc9pckOY5tw2kebWyPKfyBpa%2BguAYhcU2VZqpBUho%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAALCHLUTSVSJM5KP2XQS3RLQP2DLPANCNFSM4HVHO4BA&data=02|01|patriek.van.dorp%40microsoft.com|7b833aaecb34411aac6708d756ae2010|72f988bf86f141af91ab2d7cd011db47|1|0|637073178184450496&sdata=QTcO0BL0mZIIUCEzwvRNKxjPPg2itrnIhOW%2F9Tm2Qak%3D&reserved=0.
experiencing same @GLink worked back in early may, and no more…
@marchanddesable as a temporary workaround I manually created a service principal then passed in the details into the
az aks create
command, which worked