google-auth-library-nodejs: Sometimes failing to retrieve Application Default Credentials on GKE Autopilot in googleapis auth library

Mirror question here in Stack Overflow: https://stackoverflow.com/questions/68854640/sometimes-failing-to-retrieve-application-default-credentials-on-gke-autopilot-i

Some pods in my GKE Autopilot cluster aren’t able to grab the Application Default Credentials to call other GCP services.

I will apply a new deployment, and 1 or 2 out of the 3 pods won’t be able to authenticate using the googleapis (google-auth-library) npm package (tried with version v73.0.0 and the latest v84.0.0).

I get:

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/node_modules/google-auth-library/build/src/auth/googleauth.js:173:19)

I am using this code and retrying on failure:

const {google} = require('googleapis');

const setGoogleAuth = async () => {
    try {
        const auth = new google.auth.GoogleAuth({
            // Scopes can be specified either as an array or as a single, space-delimited string.
            scopes: ['https://www.googleapis.com/auth/cloud-platform'],
        });             
        
        // Acquire an auth client, and bind it to all future calls
        const authClient = await auth.getClient();
        google.options({auth: authClient});
    } catch (e) {
        console.error(e)
        
        //retry

        //sleep for 3 seconds
        await sleep(3000)
        

        await setGoogleAuth()
    }
    
}

Calling the metadata server manually from a troubled pod via curl --location --request POST 'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=my-gcp-endpoint' \ --header 'Metadata-Flavor: Google' \ --data-raw '{}' returns a valid token

Sometimes killing the pod and having them recreated works (using minReplicas in Horizontal Pod Autoscaler) and sometimes no matter how many times I kill the troubled pods the issue persists. Other times, I’ll redeploy and have no problems. The behaviour seems very non-deterministic.

I also tried running another node process inside the troubled pod, and did not get the error

Any help would be appreciated, thank you!

Possible related issues

https://github.com/googleapis/google-auth-library-nodejs/issues/798 https://github.com/googleapis/google-auth-library-nodejs/issues/786 https://github.com/googleapis/google-auth-library-nodejs/issues/526

Environment details

• GKE Version: 1.20.8-gke.900
• Node.js version: v10.24.1
• npm version: 6.14.12
• google-auth-library version: tried with v73.0.0 and v84.0.0 (latest)

Steps to reproduce

1. Difficult to reproduce as behaviour is non deterministic