firebase-tools: "Invalid source token" code 9 error during function deployment

[REQUIRED] Environment info

firebase-tools: 10.1.4

We are not able to upgrade to the most recent version, as it breaks the emulator suite. I saw that this issue was supposed to be fixed in the most recent release, however the emulators still fail to start up with the most recent version. Our functions config values are still not being loaded in properly when we start the emulators, so we are using 10.1.4 until further notice.

Platform: Docker image cimg/node:16.14.0 via CircleCI

[REQUIRED] Test case

Here is error output from our CircleCI build:

i  functions: updating Node.js 16 function functionName1(us-central1)...
i  functions: updating Node.js 16 function functionName2(us-central1)...
i  functions: updating Node.js 16 function functionName3(us-central1)...

Functions deploy had errors with the following functions:
	functionName1(us-central1)
	functionName2(us-central1)
	functionName3(us-central1)
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function functionName1 in region us-central1
- Error Failed to update function functionName2 in region us-central1
- Error Failed to update function functionName3 in region us-central1

Exited with code exit status 123
CircleCI received exit code 123

When looking at the function logs, I have seen a variety of vague errors that I cannot find the cause of. Here is one of these errors:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 14,
      "message": "The service has encountered an error during container import. Please try again later"
    },
    "authenticationInfo": {
      "principalEmail": "REDACTED"
    },
    "serviceName": "cloudfunctions.googleapis.com",
    "methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
    "resourceName": "REDACTED"
  },
  ...
  "severity": "ERROR",
  ...
}

And here is another:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {
      "code": 9,
      "message": "Invalid source token"
    },
    "authenticationInfo": {
      "principalEmail": "REDACTED"
    },
    "serviceName": "cloudfunctions.googleapis.com",
    "methodName": "google.cloud.functions.v1.CloudFunctionsService.UpdateFunction",
    "resourceName": "REDACTED"
  },
  "severity": "ERROR",
  ...
}

These logs are preceded by about a dozen protoPayload.methodName: google.longrunning.Operations.GetOperation calls in the function logs. These errors appear consistently and reproducibly.

[REQUIRED] Steps to reproduce

Run a function deployment from CircleCI using Firebase version 9.23.3 or later (this is the first version we encountered issues with, and continued encountering issues after upgrading) and Node version 16.14.0.

Deploying from the command line works fine.

[REQUIRED] Expected behavior

The CircleCI deployment succeeds and all functions update successfully as they did before the firebase-tools upgrade.

[REQUIRED] Actual behavior

The deployment fails with the errors above.

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 2
Comments: 31 (8 by maintainers)

Most upvoted comments

@trex-quo Thanks for the feedback. 50% success rate is terrible, and we should strive to do better. We’ll work on tuning our parameters to deal w/ quota issues better.

In the immediate terms, it sounds like breaking up deploys as you’ve done before but NOT running them in parallel might be most reliable thing to do.

taeold on Mar 16, 2022

Still the issue even in 11.6.1 (node 14.21):

Failed 8 deployments in a row. Tired of retrying.

I have 29 functions and last 4 always fail now.

n-sviridenko on Nov 30, 2022

@tojtoj That’s pretty much what was happening to me. Upgrading to 11.4.0 fixed the “Invalid source token” error. ~~It also brought other error (https://github.com/firebase/firebase-tools/issues/4730), but I fixed it with https://npm.im/patch-package.~~

EDIT: https://github.com/firebase/firebase-tools/issues/4730 is fixed in 11.4.2.

gustavopch on Aug 3, 2022

Here is an instance where a “Quota exceeded” error failed the deployment. Since merging the change into our CI that deploys all functions in a single command, the success rate seems about 50%. Here is the full (redacted) output of the deployment.


Now using project project-name

=== Deploying to 'project-name'...

i  deploying functions
Running command: predeploy_script.js
Currently in project-name. Setting flags and deploying...

> lint
> eslint .
> build

✔  functions: Finished running predeploy script.
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (3.12 MB) for uploading
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: functions folder uploaded successfully

The following functions are found in your project but do not exist in your local source code:
	functionName(us-central1)
	(~10 more functions listed)

If you are renaming a function or changing its region, it is recommended that you create the new function first before deleting the old one to prevent event loss. For more info, visit https://firebase.google.com/docs/functions/manage-functions#modify

? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) ? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "n? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "nn? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. (y/N) "nn"? Would you like to proceed with deletion? Selecting no will continue the rest o
f the deployments. No

i  functions: updating Node.js 16 function functionName(us-central1)...
(repeated for ~70 more functions)

Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
(repeated for ~25 more functions)

⚠  functions: got "Quota Exceeded" error while trying to update projects/project-name/locations/us-central1/functions/functionName. Waiting to retry...
(repeated for ~50 more functions

✔  functions[functionName2(us-central1)] Successful update operation.
(repeated for ~20 more functions)

Functions deploy had errors with the following functions:
	functionName3(us-central1)
	(repeated for ~30 more functions)

Function URL (functionName4(us-central1)): *functionURL*
(repeated for ~50 more functions)

i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function functionName in region us-central1
(repeated for ~50 more functions)

Exited with code exit status 2
CircleCI received exit code 2

trex-quo on Mar 15, 2022

@taeold

I did try a single firebase deploy --only functions command a couple weeks back, but ran into a separate error which prevented the full deployment from going through. It is the same issue as on this thread:

https://github.com/firebase/firebase-tools/issues/3919#issuecomment-979788970

One comment on the thread by @abeisgoat says:

Hey we’re aware of issues when deploying over 100 functions, it’s best to deploy the functions you changed specifically instead of the whole lot when you’re at this scale.

which seems to suggest that it is known that deploying all functions in a single command won’t work.

That being said, I did just try another full deployment now with ~110 functions and it appeared to go through successfully. I have deleted some unused functions since the last time I tried this (it was probably ~140 functions at the time). Since our pipeline is failing now anyway, I will try replacing our CI system with a single firebase deploy --only functions command and monitor the rate at which it fails and succeeds over the coming weeks.

trex-quo on Mar 14, 2022