azure-sdk-for-js: Frequent 429 throttling using MSI Credential for CosmosDB

Package Name: @azure/identity
Package Version: 1.5.1
Operating system: Ubuntu 16.01
nodejs
- version: 12.21.0
browser
- name/version:
typescript
- version: 4.3.2
Is the bug related to documentation in
- README.md
- source code documentation
- SDK API docs on https://docs.microsoft.com

Describe the bug My team is attempting to migrate to the new preview feature to use AAD credentials to access CosmosDB through CosmosClient. Previously, we had been fetching the access key with an MSI token and the listkeys api endpoint, similar to this example code.

After migrating to using the aadCredentials option instead, we are experiencing frequent 429 responses from the MSI server when trying to perform a cosmos operation. #8950 looks like a similar problem, but the suggested solution of performing an initial MSI call to make sure the token is fetched and cached has not fixed the issue for us. We’re using the same Credential type (DefaultAzureCredential) that we used originally to get the API token.

My understanding is that the credential should retry getToken and handle token caching and refresh without any interference from the user. Is this correct, or do we need to configure something differently on our end? Does CosmosClient make frequent calls to the MSI server when using aadCredentials?

To Reproduce Since this runs in our web service, it is difficult to determine the specific scenario that will reproduce the issue locally. Let me know if more details are needed about our specific scenario.

Expected behavior using aadCredentials should not throw 429 in a scenario where calling getToken manually and fetching a key from the API does not.

Screenshots

Additional context It may be worth mentioning that the code in question is running in a web service where several requests may happen concurrently, but again, we have not had this problem prior to attempting to use aadCredential.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 22 (19 by maintainers)

Commits related to this issue

CodeGen from PR 17120 in Azure/azure-rest-api-specs Compute 2021-11-01 Release (#17120) * compute folder with pre-population * fix examples folder * fix an example file * update compute to m... — committed to AzureSDKAutomation/azure-sdk-for-js by deleted user 2 years ago

Most upvoted comments

i’ll merge this once I get it passing if it looks good and we can try another dev build: https://github.com/Azure/azure-sdk-for-js/pull/17289

zfoster on Aug 26, 2021

I’ll compare the header before and after the change and post the results here tomorrow

zfoster on Aug 26, 2021

Upgrading to 3.14.0 solved the issue. Thank you!

pownkel on Sep 3, 2021

well, you can in fact alter how the token is assigned to the request by specifying a custom authorizeRequest callback: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/core/core-rest-pipeline/src/policies/bearerTokenAuthenticationPolicy.ts#L61

here’s the default one: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/core/core-rest-pipeline/src/policies/bearerTokenAuthenticationPolicy.ts#L94-L105

Here’s an example of how we customize it in one of our tests: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/core/core-rest-pipeline/test/node/bearerTokenAuthenticationPolicyChallenge.spec.ts#L169-L176

I believe it should look more like this for you:

bearerTokenAuthenticationPolicy({
  challengeCallbacks: {
    async authorizeRequest({ request, getAccessToken }) {
      const token = await getAccessToken([`scope`], {});
      const AUTH_PREFIX = `type=aad&ver=1.0&sig=`;
      const authorizationToken = `${AUTH_PREFIX}${token}`;
      request.headers.set("Authorization", `Bearer ${authorizationToken}`);
    },
  }
});

Let me know if that helps!

sadasant on Aug 26, 2021

ah the token format is definitely the issue, here’s how we have it formatted. I didn’t realize it’s sent as bearer:

https://github.com/Azure/cosmos-explorer/blob/7d0be7d355b17ddd2db79b39a609350c61dd671b/src/Common/CosmosClient.ts#L14-L18

zfoster on Aug 26, 2021