azure-sdk-for-js: Managed Identity - DefaultAzureCredential - RestError: [TokenExpired] Error validating token: 'S2S12086'

Package Name:
@azure/identity
Package Version:
^2.0.0
Operating system:
Linux
nodejs
- version: 16.x
browser
- name/version:
typescript
- version:
Is the bug related to documentation in
- README.md
- source code documentation
- SDK API docs on https://docs.microsoft.com

Describe the bug

Hello

I have a Node.js application running in App Service that needs to fetch secrets from a KeyVault. This App Service has a Managed Identity. Here’s the relevant pseudo code :

const secretClient = new SecretClient(this.keyVaultUrl, new DefaultAzureCredential());
const secret = (await secretClient.getSecret(secretName))?.value;

This has been working great for months, but suddently about 10 days ago, our app wasn’t able to fetch secrets anymore. A first basic look at the logs, showed this error message, coming from the underlying library:

RestError: [TokenExpired] Error validating token: 'S2S12086'

We’re using DefaultAzureCredential, which in our case instantiate a ManagedIdentityCredential in the end. So it looks like that there’s an issue with the token expiration date received by the App Service by the internal Azure issuer.

To better understand what’s happening under the hood, I enabled the maximum verbosity with AZURE_LOG_LEVEL=verbose environment variable.

Then, one particular line looks quite suspicious at the end of the token acquisition dance :

azure:identity:info IdentityClient: [http://172.16.2.7:8081/msi/token?resource=https%3A%2F%2Fvault.azure.net&api-version=2019-08-01] token acquired, expires on NaN

It corresponds to the following line : https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/identity/identity/src/client/identityClient.ts#L140

The NaN at the end of the log line (which is token.accessToken.expiresOnTimestamp) looks quite surprising and I think it may be related to the problem we experiencing.

Do you confirm this intuition? Thanks.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 20 (11 by maintainers)

Commits related to this issue

work-bus: upgrade to @azure/identity 3.1.1 After recent refactoring of the github-reflector to persist connections to Azure resources, we were seeing access to these objects fail. Others had reporte... — committed to microsoft/msr-morello-automation by nwf-msr 2 years ago
CodeGen from PR 22722 in Azure/azure-rest-api-specs Updated readmes (#22722) — committed to azure-sdk/azure-sdk-for-js by deleted user a year ago

Most upvoted comments

Hi again patient friends, I would like to update this thread with the news that my fix was released as part of @azure/identity 3.0.0 yesterday. 🎉

I believe this should hopefully resolve the issue, so in my optimism I am going to resolve but please let me know if you are still having issues with token expiration!

xirzec on Sep 20, 2022

Just lost 2 days troubleshooting this 😔

As reported by multiple people above, on version 2.1.0 sometimes the expiresOnTimestamp is NaN, which caused our internal refresh logic to fail and never refresh it, so after 24h the token became invalid and we were still trying to use it.

For context, we’re using App Service with user-assigned managed identity.

goenning on Aug 30, 2022

I ran a test over the weekend. Just a simple timer triggered Function that writes to Cosmos DB. I had three versions of this Function App: "@azure/identity": "^2.0.5", "@azure/identity": "^2.1.0", and "@azure/identity": "^3.0.0-beta.1"

const { DefaultAzureCredential } = require("@azure/identity");
const { CosmosClient } = require("@azure/cosmos")

const client = new CosmosClient({
    endpoint: process.env["CosmosDb__accountEndpoint"],
    aadCredentials: new DefaultAzureCredential(),
  });
const database = client.database("a")
const container = database.container("a")

module.exports = async function (context, myTimer) {
    var timeStamp = new Date().toISOString();

    const id = "1";
    const partitionKey = "205";
    const res = await container.items.upsert({
                    id,
                    partitionKey,
                    timeStamp});
};

v2.1.0 as well as v3.0.0-beta.1 failed after ~24h. Not so v2.0.5. It still runs fine after more than 48h.

mfehlhaber on Aug 29, 2022

Hi.

After a bit more investigation, the problem appeared because our dependency on @azure/identity was a bit loose semver-wise. Meaning that everything was fine with version 2.0.5 and problem started to appear when 2.1.0 was released.

My understanding after looking at the log message is that the library is working well with a token returned from an endpoint using the 2017 API but something wrong is happening as soon as it starts interacting with an endpoint the 2019 API.

Delapouite on Aug 1, 2022