google-auth-library-nodejs: Google Cloud Function - Error: Could not load the default credentials.

There’s an internal issue with the GCF folks that has been updated throughout the day, we’re trying to get the root cause of the timeouts that occur attempting to connect to the metadata server (this is what’s in turn resulting in the credentials issue).

A potential workaround for folks, would be to create a service account, rather than relying on the default credentials:

https://cloud.google.com/docs/authentication/getting-started

☝️ this requires that the credentials are available in a file on disk, so you would need to either use the API to deploy your project, or use the file upload option (rather than the inline editor).

At which point you would set your credentials to GOOGLE_APPLICATION_CREDENTIALS=./my-service-account.son.

I understand this workaround is suboptimal, and we are continuing to dig into things on our end.

@bcoe This issue remains unsolved and I face the error daily. Do you have any update(s) regarding the progress on this issue?

I’m having the same problem with various libraries inside cloud functions.

This example is listening for a bucket onFinalize event and sending a single http task to a cloud tasks queue with the payload.

It seems that my function works fine when its first deployed. Then after some time, (perhaps) after it scales to zero and then is re-triggered it always fails until I redeploy it.

I’m using the nodejs10 runtime with "@google-cloud/tasks": "^1.4.0".

Code

const { v2beta3 } = require('@google-cloud/tasks')
const client = new v2beta3.CloudTasksClient()

const queue = body => {
  return client.createTask({
    parent: process.env.QUEUE_URL,
    task: {
      httpRequest: {
        httpMethod: 'POST',
        url: process.env.TASK_URL,
        headers: { 'Content-Type': 'application/json' },
        body: Buffer.from(JSON.stringify(body)),
        oidcToken: { serviceAccountEmail: process.env.SERVICE_ACCOUNT_EMAIL }
      }
    }
  })
}

exports.default = async file => {
  await queue(file)
  console.info(`DONE: ${file.name} queued`)
}

Actually, I’ve just seen a new occurrence of the problem.

At 14:12:30.749, I got the error reported above: Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information. at GoogleAuth.getApplicationDefaultAsync (/srv/node_modules/google-gax/node_modules/google-auth-library/build/src/auth/googleauth.js:160:19) at <anonymous> at process._tickDomainCallback (internal/process/next_tick.js:229:7)

And before that, in the traces, I saw:

At 14:11:11.936, the trace:

{ FetchError: network timeout at: http://169.254.169.254/computeMetadata/v1/instance
    at Timeout.<anonymous> (/srv/node_modules/node-fetch/lib/index.js:1448:13)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
  message: 'network timeout at: http://169.254.169.254/computeMetadata/v1/instance',
  type: 'request-timeout',
  config: 
   { url: 'http://169.254.169.254/computeMetadata/v1/instance',
     headers: { 'Metadata-Flavor': 'Google' },
     retryConfig: 
      { noResponseRetries: 0,
        currentRetryAttempt: 0,
        retry: 3,
        httpMethodsToRetry: [Array],
        statusCodesToRetry: [Array] },
     responseType: 'text',
     timeout: 3000,
     params: {},
     paramsSerializer: [Function: paramsSerializer],
     validateStatus: [Function: validateStatus],
     method: 'GET' } }

And 400ms later at 14:11:12.336, I got:

{ FetchError: network timeout at: http://169.254.169.254/computeMetadata/v1/instance
    at Timeout.<anonymous> (/srv/node_modules/node-fetch/lib/index.js:1448:13)
    at ontimeout (timers.js:498:11)
    at tryOnTimeout (timers.js:323:5)
    at Timer.listOnTimeout (timers.js:290:5)
  message: 'network timeout at: http://169.254.169.254/computeMetadata/v1/instance',
  type: 'request-timeout',
  config: 
   { url: 'http://169.254.169.254/computeMetadata/v1/instance',
     headers: { 'Metadata-Flavor': 'Google' },
     retryConfig: 
      { noResponseRetries: 0,
        currentRetryAttempt: 0,
        retry: 3,
        httpMethodsToRetry: [Array],
        statusCodesToRetry: [Array] },
     responseType: 'text',
     timeout: 3000,
     params: {},
     paramsSerializer: [Function: paramsSerializer],
     validateStatus: [Function: validateStatus],
     method: 'GET' } }

Both seemed to use the IP address of the metadata server, and not the DNS name.

Note, the workaround of specify an explicit service account, using GOOGLE_APPLICATION_CREDENTIALS should do the trick for the time being as a workaround.

Hi @bcoe would that be an env variable at deploytime or a file in the root of the function? Thanks!

@seriousManual I used this workaround and haven’t had any errors since. I uploaded the secret in the root of the function and then declared the variable at the start of my code:

process.env.GOOGLE_APPLICATION_CREDENTIALS = './my-secret.json';

Firestore, BigQuery, Debugger, …

Given the stack traces, the error observed originates here: https://github.com/googleapis/google-auth-library-nodejs/blob/944e2aa62a61c253ba153f49590d7416585c64eb/src/auth/googleauth.ts#L291-L296

As you can see, it is thrown if and only if the value of isGCE variable is falsy. The value is a result of a call of _checkIsGCE function: https://github.com/googleapis/google-auth-library-nodejs/blob/944e2aa62a61c253ba153f49590d7416585c64eb/src/auth/googleauth.ts#L311-L316

This function in turn calls isAvailable function from google-metadata library: https://github.com/googleapis/gcp-metadata/blob/25bc11657001cb6b3807543377d74bafe126ea62/src/index.ts#L121-L142

As you can see, it depends on metadataAccessor function: https://github.com/googleapis/gcp-metadata/blob/25bc11657001cb6b3807543377d74bafe126ea62/src/index.ts#L49

This function makes an HTTP request to http://169.254.169.254/computeMetadata/v1/ here: https://github.com/googleapis/gcp-metadata/blob/25bc11657001cb6b3807543377d74bafe126ea62/src/index.ts#L66

I see no other way for this error to occur other than a requests to this service fail.

Yeah, I did the same.

I abandoned using the default creds that come with cloud functions since they’re the root cause of the issue (being inconsistently available). Uploading a service account and directly pointing to it works.

I still think the awesome G-team should find the root cause and release a patch

Just to add on this, we’re seeing this error as well in the context of logging-winston (https://www.npmjs.com/package/@google-cloud/logging-winston).

This happens almost everytime the cloud function is redeployed (e.g. after an update). The function is running but logging does not work. I’m writing almost here, because from time to time it is actually working, but not consistently.

@vmal @kthaas @omersadika @calclavia, we believe we’ve isolated this problem to creating a client in the global scope outside of your function see: function execution lifetime.

If your code looks like this:

const {CloudTasksClient} = require('@google-cloud/task');
// CloudTasksClient, could be Storage, or Vision, or Speech.
const task = new CloudTasksClient();
export.helloWorld = (req, res) => {
  res.send('hello world');
}

Instead write:

const {CloudTasksClient} = require('@google-cloud/task');
// CloudTasksClient, could be Storage, or Vision, or Speech.
let task;
export.helloWorld = async (req, res) => {
  if (!task) {
    task = new CloudTasksClient();
  }
  await task.doSomething(); // or storage.doSomething(), etc.
  res.send('hello world');
}

There is work in motion to shield folks from this issue. But, as described in the execution timeline, the core of the problem is that asynchronous work cannot happen outside of the request cycle of your function.

I’m getting the same issue for firebase functions. It’s randomly happening sometimes after I deploy. The way to solve it is by redeploying the function / deleting and deploying until it’s working. This is destroying productivity for my team, we’re spending hours on dealing with it.

Try export GOOGLE_APPLICATION_CREDENTIALS=~/.config/path-to-credential.json

@glaforge @Cyganieszka @jamesbenns @BrodaNoel @psyh, I apologize for how frustrating this intermittent outage has been (and for hiding your issue @jamesearl).

I’m in the process of writing up a document, to communicate the severity of this problem to peers at Google. I have a favor to ask, could I ask folks to run their service with the environment variable:

DEBUG_AUTH=true

And to send me the error that you see in logs, either by responding in thread or by emailing (bencoe [at] google.com). This will hopefully give me the information I need to get more eyeballs on the problem.

@jamesbenns @jketcham @adamworrall, and others, could I bother you to try setting the environment variable DETECT_GCP_RETRIES to 3? This will make the library retry when detecting whether the environment is GCP.

Along with this, could you also set the environment variable DEBUG_AUTH to true, this will provide more detailed information about the error that occurred detecting the environment (I would love to see this detailed information).

Came here to leave a suggestion: you might have promise callbacks in your code not managed properly (using await). I faced the same auth error as above, but then found this answer https://stackoverflow.com/a/58779000/8500237 ; after rechecking my code, I found that awaiting the promises fixed also the functions errors 😃 Hope it helps someone else.

@aldobaie Not the case.

It’s just a request timeout, when it happens. Seems like the SDK can’t each one of the credentials endpoints ( eg. 169.254.169.254 ).

Given the fact that it happens randomly ( for the same code snippets ) means it’s not code-oriented.

The workaround (until they sort it) is to load your service account credentials manually like this:

const credentials = require('./credentials.json')
admin.initializeApp({
	credential: admin.credential.cert(credentials),
	databaseURL: 'https://PROJECT_ID.firebaseio.com'
})

We are also having this issue. We have about 65 Firebase / Google cloud functions in deployment. Our services will randomly fail at some points with this failure to load default credentials issue and the only remediation we have found is a redeployment.

@bcoe Update: Since making the change discussed above, I have not experienced the error. 🙌

I’m encountering same error. In my case I do Promise.all(list_of_promises), where each promise does http request(by request-promise). It works as expected if list_of_promises length is ~10, it fails with topic’s error if array is bigger. I double checked if I handle all promises correctly, but I’m not a JS programmer, so I keep in mind that could be my fault…

I’m also using Firebase Cloud functions and experiencing this issue only for functions connecting to the Vision API. I have to just keep redeploying until it miraculously works (about 1 in 10 times)! 🤬

@martinatcode are you using firebase functions as well?

Same here. The error is occurring in recently updated Firebase Cloud Functions that access Cloud Firestore or the Firebase Realtime Database using the admin SDK.

I was observing the issue on the last week of 2019, a function that worked OK a while before has begun failing with the default credentials error on a Firestore request, but no changes has been made. So I just re-deployed the function with the same source code and it just helped.

@timhj I doubt that it relates to specific APIs, and am pretty convinced that it is caused by intermittent issues accessing the metadata server that we use to detect environment in google cloud functions. There’s another PR I’ll merge Monday, which adds additional caching:

https://github.com/googleapis/google-auth-library-nodejs/pull/858

My hope is that this will further improve the situation.

The function was deployed today.

It always happens. This can be avoided by setting an environment variable.

Yes, I get it only when do highly concurrent firestore processing from firebase functions (reading/writing non-overlapping documents from different collections)

@moblizeit @stari4ek are you both using Cloud Functions? I have a recommendation I am going to make an effort to deploy this week.

The workaround of referencing the service account directly should address the issue, but we are making efforts to address the underlying issues.

Good workaround!

Though the good folks on the team need to dig into the root cause since their software is inconsistent at scale, which means there’s a deeper root cause to discover and patch 😃

Have not seen it in a while, but the project does not have high activity at this point.

Edit: Scratch that, just did some tests and got it again.

Sure thing, I can email Firebase support. Thanks

@Tebro I see a few internal threads regarding timeout issues on the metadata server, there was one rollout today, but it looks like the issue is still periodically happening.

I will continue to keep this thread updated.

Note, the workaround of specify an explicit service account, using GOOGLE_APPLICATION_CREDENTIALS should do the trick for the time being as a workaround.