azure-sdk-for-net: KeyVault - SecretClient.GetSecret(string key) return Status: 401 (Unauthorized)

Describe the bug When I am calling SecretClient.GetSecret(key) I am sometimes getting: Azure.RequestFailedException: Status: 401 (Unauthorized). This method is usually called multiple times in parallel using single instance of SecretClient. The constructor I use for secret client: new SecretClient(new Uri(keyVaultUrl), new ManagedIdentityCredential()); Example from my logs says that it can fail 3 times for 6 concurrent requests.

Expected behavior The method should not throw.

Actual behavior (include Exception or Stack Trace)

Azure.RequestFailedException: 
Status: 401 (Unauthorized)

Content:
{""error"":{""code"":""Unauthorized"",""message"":""Request is missing a Bearer or PoP token.""}}

Headers:
Cache-Control: no-cache
Pragma: no-cache
Server: Microsoft-IIS/10.0
WWW-Authenticate: Bearer authorization=""https://login.windows.net/d37240c9-786e-4171-a9f4-37cfd2adf51f"", resource=""https://vault.azure.net""
x-ms-keyvault-region: northeurope
x-ms-request-id: 1f648564-3db0-41b3-bba3-7fadec5198cb
x-ms-keyvault-service-version: 1.1.0.891
x-ms-keyvault-network-info: addr=40.113.80.190;act_addr_fam=InterNetwork;
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Strict-Transport-Security: max-age=31536000;includeSubDomains
X-Content-Type-Options: nosniff
Date: Fri, 31 Jan 2020 09:07:17 GMT
Content-Length: 87
Content-Type: application/json; charset=utf-8
Expires: -1

   at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest[TResult](RequestMethod method, Func`1 resultFactory, CancellationToken cancellationToken, String[] path)
   at Azure.Security.KeyVault.Secrets.SecretClient.GetSecret(String name, String version, CancellationToken cancellationToken)

To Reproduce Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

Call GetSecret in parralel (for same resource) with KeyVaultTestClient singleton

public class KeyVaultTestClient
    {
        private readonly ConcurrentDictionary<string, string> _cachedSecrets = new ConcurrentDictionary<string, string>();
        private readonly SecretClient _secretClient;

        public KeyVaultTestClient(string keyVaultUrl)
        {
            _secretClient = new SecretClient(new Uri(keyVaultUrl), new ManagedIdentityCredential());
        }

        public string GetSecret(string key)
        {
            if (_cachedSecrets.ContainsKey(key))
            {
                return _cachedSecrets[key];
            }

            var secret = _secretClient.GetSecret(key)?.Value?.Value; // GetSecret throws sometimes

            if (string.IsNullOrEmpty(secret))
            {
                throw new KeyNotFoundException($"Secret=[{key}]");
            }

            _cachedSecrets[key] = secret;
            return secret;
        }
    }

Environment:

• Name and version of the Library package used: Azure.Security.KeyVault.Secrets 4.0.1 & Azure.Identity 1.1.0
• Hosting platform or OS and .NET runtime version (dotnet --info output for .NET Core projects): e.g. Azure AppService .NET Core 3.1 - I do not see the problem on my other app in Azure which is using .NET Framework 4.7.2