azure-sdk-for-net: [BUG] Azure.Identity 1.5.0 freezes up ChainedTokenCredential with ManagedIdentityCredential listed first in local dev

Hi @christothes

I can provide my details.

My service is using DefaultAzureCredential without any other options and running the code on my laptop (local development), using VS 2022.

I’ve done az login in the command line to cache my personal credentials to access Azure Resources. If I run it from the command line using dotnet run (not using VS 2022 at all), I get the same behaviour.

I observed the delay with connections to Azure Key Vault and Azure Service Bus.

Following is sample code connecting to KeyVault. You can see from the logs that it takes 10 seconds when ManagedIdentityCredential is not excluded vs 3 seconds when ManagedIdentityCredential is excluded.

Sample code 1

using Azure.Identity;
using Azure.Security.KeyVault.Secrets;

Console.WriteLine($"{DateTime.Now} Creating client for key vault");
var secretClient = new SecretClient(new Uri("https://mykeyvault.vault.azure.net/"), new DefaultAzureCredential());
Console.WriteLine($"{DateTime.Now} Getting entry from key vault");
var v1 = secretClient.GetSecretAsync("key1").Result.Value;
Console.WriteLine($"{DateTime.Now} Done");
Console.WriteLine($"{DateTime.Now} Getting another entry from key vault");
var v2 = secretClient.GetSecretAsync("key2").Result.Value;
Console.WriteLine($"{DateTime.Now} Done");

Logs (10 seconds to get the first key)::

19/10/2022 00:37:41 Creating client for key vault
19/10/2022 00:37:41 Getting entry from key vault
19/10/2022 00:37:51 Done
19/10/2022 00:37:51 Getting another entry from key vault
19/10/2022 00:37:51 Done

Sample code 2 (ExcludeManagedIdentityCredential = true)

using Azure.Identity;
using Azure.Security.KeyVault.Secrets;

Console.WriteLine($"{DateTime.Now} Creating client for key vault");
var secretClient = new SecretClient(new Uri("https://mykeyvault.vault.azure.net/"), new DefaultAzureCredential(
    new DefaultAzureCredentialOptions()
    {
        ExcludeManagedIdentityCredential = true
    }));
Console.WriteLine($"{DateTime.Now} Getting entry from key vault");
var v1 = secretClient.GetSecretAsync("key1").Result.Value;
Console.WriteLine($"{DateTime.Now} Done");
Console.WriteLine($"{DateTime.Now} Getting another entry from key vault");
var v2 = secretClient.GetSecretAsync("key2").Result.Value;
Console.WriteLine($"{DateTime.Now} Done");

Logs (3 seconds to get first key):

19/10/2022 00:40:26 Creating client for key vault
19/10/2022 00:40:26 Getting entry from key vault
19/10/2022 00:40:29 Done
19/10/2022 00:40:29 Getting another entry from key vault
19/10/2022 00:40:29 Done

Packages in my project (same behaviour with Azure.Identity 1.8.0-beta.1 and 1.9.0-beta.1):

    <PackageReference Include="Azure.Identity" Version="1.7.0" />
    <PackageReference Include="Azure.Security.KeyVault.Secrets" Version="4.4.0" />

System Info

C:\Users\helder.sousa>dotnet --version
6.0.400

C:\Users\helder.sousa>systeminfo
OS Name:                   Microsoft Windows 10 Enterprise
OS Version:                10.0.19044 N/A Build 19044
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Member Workstation
OS Build Type:             Multiprocessor Free
System Manufacturer:       Dell Inc.
System Model:              Latitude 5420
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 140 Stepping 1 GenuineIntel ~1805 Mhz
BIOS Version:              Dell Inc. 1.19.0, 16/06/2022
Windows Directory:         C:\WINDOWS
System Directory:          C:\WINDOWS\system32
Boot Device:               \Device\HarddiskVolume1
Total Physical Memory:     32,497 MB
Available Physical Memory: 8,189 MB
Virtual Memory: Max Size:  46,833 MB
Virtual Memory: Available: 14,541 MB
Virtual Memory: In Use:    32,292 MB
Hotfix(s):                 20 Hotfix(s) Installed.
                           [01]: KB5017022
                           [02]: KB4562830
                           [03]: KB4570334
                           [04]: KB4577266
                           [05]: KB4577586
                           [06]: KB4580325
                           [07]: KB5000736
                           [08]: KB5003791
                           [09]: KB5012170
                           [10]: KB5017308
                           [11]: KB5006753
                           [12]: KB5007273
                           [13]: KB5011352
                           [14]: KB5011651
                           [15]: KB5014032
                           [16]: KB5014035
                           [17]: KB5014671
                           [18]: KB5015895
                           [19]: KB5016705
                           [20]: KB5005699


C:\Users\helder.sousa> wmic cpu get caption, name, deviceid, numberofcores, maxclockspeed, status
Caption                                DeviceID  MaxClockSpeed  Name                                            NumberOfCores  Status
Intel64 Family 6 Model 140 Stepping 1  CPU0      1805           11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz  4              OK

Hi @heldersousa-planetpayment - I think the reason that the distinct DefaultAzureCredential works relatively the same as the reused example is that, under the covers, when you don’t pass any options to DefaultAzureCredential you actually get a static singleton each time.

@schaabs I just want to report my findings after implementing the changes suggested.

Even with the NetworkTimeout set to 1s, or simply using DefaultAzureCredential which has that by default, it still takes considerably longer than 1.4.1 to time out ManagedIdentityCredential. I count approximately 10 seconds, long enough to trigger the “unable to connect to web server” warning in Visual Studio. So I assume that the 1s timeout only comes into play after some initial struggle which itself takes many seconds.

As is right now with the package, I’ll have to revert back to 1.4.1. If this issue doesn’t eventually get resolved, I’m thinking I can probably check the environment (via something like IWebHostEnvironment) and separate the credential per environment instead of chaining them like the old model. It can be a bit of a pain though because we use such credentials extensively in our app (we aim for 100% MI whenever possible).

Hopefully y’all can come up with a resolution to restore the performance of the older package without sacrificing reliability.

By my naïve imagination, wouldn’t there be at least some kind of reliable “traits”/environment variables that can help decide “are we in Azure? (the only place MI is relevant)” before even attempting to call the MI endpoint (169.254.169.254?), and if not, skip the MI credential altogether?

@mikequ-taggysoft

are you basically suggesting that ManagedIdentityCredential can sometimes have a longer latency (potentially >= 1s) while running inside Azure, despite it using Microsoft-managed private network?

No, a 1 second timeout should be safe since the managed identity endpoint is generally a non-addressable local endpoint. We did some extensive performance benchmarking on various managed identity hosts and found this to be an acceptable limit. What I was trying to suggest is that setting NetworkTimeout to a considerably shorter amount of time, say 300ms, will further speed up your ChainedTokenCredential in your development enviornment, but depending on the host and your application you might see some artificial timeouts when deployed. Sorry for the confusion.

BTW, the only reason I’m using ChainedTokenCredential over DefaultAzureCredential is because the latter places VisualStudioCredential in a higher priority, and back then I found it to be very slow. AzureCliCredential on the other hand is always very fast.

You can also skip the VisualStudioCredential with DefaultAzureCredential by setting this to true: https://github.com/Azure/azure-sdk-for-net/blob/f993f9d5a6062a04feedd7a72ac71dcb0c7ce77f/sdk/identity/Azure.Identity/src/DefaultAzureCredentialOptions.cs#L98

Now after seeing your post, I did test the VisualStudioCredential again (running VS 2022 RC1) and it appears fast so far. Has the performance of this credential been tuned and can be expected to be consistently speedy nowadays?

There have been no changes in the SDK but perhaps there were some improvements shipped with VS’s token utility.

@mikequ-taggysoft sorry you’re running into this trouble. This delay was introduced by this refactoring to make the ManagedIdentityCredential endpoint discovery more reliable. Unfortunately, depending on your systems network configuration, failures may not be immediate, and fail due to timeout after the full duration of RetryOptions.NetworkTimeout which I believe defaults to 90 seconcds.

After discovering this timeout behavior in some dev configurations we added this fix which limits the initial network connection timeout when the ManagedIdentityCredential is used in the DefaultAzureCredential. Unfortunately this only fixes the issue in ManagedIdentityCredential when used in the DefaultAzureCredential chain, not when used with a custom chain with the ChainedTokenCredential as in your case.

You should be able to work around this issue by configuring the NetworkTimeout on the ManagedIdentityCredential to limit the time it will wait on a network response. I would start by configuring it to 1 second as we have done in the DefaultAzureCredential, but you can experiment to see what values work best for your development environment and deployed environment. Keep in mind setting too short NetworkTimeout durations will lead to artificial network timeouts when deployed to a managed identity enabled host, causing the ManagedIdentityCredential to throw a CredentialUnavailableException. Below is an example of how you can update your code to configure the NetworkTimeout.

   var miCredOptions = new TokenCredentialOptions { Retry = { NetworkTimeout = TimeSpan.FromSeconds(1) }};

   var tokenCred = new ChainedTokenCredential(new ManagedIdentityCredential(options: miCredOptions), new AzureCliCredential());

We’re currently exploring options of how to fix this long delay in some development environments when using ManagedIdentityCredential inside a ChainedTokenCredential, but hopefully this work-around will unblock you in the meantime. Please let me know if you have any trouble with the work-around. I’ll update this issue once we have a better idea of what the fix might be, and when it will be available.