firebase-tools: Can't deploy all functions consistently. Fails most of the time.

We will prioritize looking into this soon. There will be a three pronged strategy:

1. Fixing the “invalid source token” problem, which occurs when function deploys take a very long time
2. Looking at tuning the rate limiter to avoid quota errors
3. Improving deploy code so that it can recognize and skip functions that don’t need redeployment

Hopefully this will help you all when we can get these improvements ready.

@taeold @inlined can we reopen pls? and any updates or recommended best practices here?

Splitting up the codebase to take advantage of “codebases” is really clunky, especially when a team is just doing it for deploy optimization… leads to having to manage multiple package.json and yarn.lock files, changing how you build things, etc.

So updating to the latest version of firebase CLI worked for me now it is saying Skipping the deploy of unchanged functions with experimental support for skipdeployingnoopfunctions and it skips as well which is great. But the whole point of this functionality is if I have 100 functions and I make changes to 1 function then running the deploy command will skip 99 functions and deploy only the modified function.

So what happened is

1. I ran the command firebase deploy --only functions the first time after updating CLI and it deployed all the functions (good)
2. I ran the command again and it skipped all the functions. (Great)
3. I made a change to a function and ran the command again and it started deploying all the functions again (bad)

What am I doing wrong?

Bad bot. Can we reopen please? 😀

@alexlouden Thanks for an awesome writeup. The migration pains you described are really insightful and I’ll take this back to the team.

Codebase is just a start and we want to continue to explore we can better support projects with many many function. It sounds ESBuild is an interesting approach we can explore - we’ve also had internal conversation about whether natively supporting npm or yarn workspaces would help with folk’s need.

One other direction we’ve brainstormed (specifically with v2 functions) is an idea of “eventing monolith” - i.e. a single Cloud Function instance that’s responsible for processing multiple event sources. Consolidating functions that are called sparesley would reduce number of cold start and possibly save on infra cost.

I’ll go back to the team to look into skip-deployment issue. Thanks for the heads up.

We’ve currently launched (but not well documented or added wizzard support for) codebases. With codebases you can have multiple directories of Cloud Functions. Simply create a firebase.json fragment that looks a bit like:

"functions" : [
  {
    "source": "functions",
    "codebase": "main"
  }, {
    "source": "otherfunctions",
    "codebase": "other"
  }
]

With this, you can now call firebase deploy functions:main or firebase deploy functions:other. Next, we’re working on recognizing that a codebase hasn’t been modified since last deployment and we’ll skip those functions altogether.

@inlined Sorry, but I don’t see how introducing codebases aims to solve this issue (what if a single codebase has more functions than the quota limit allows?). While it might become useful in some cases it certainly won’t be helpful for me.

Wouldn’t it be better if there was an option to deploy only modified functions? (--only-changed?) Usually a single push only affects a few functions so deploying only those would help.

Since there haven’t been any recent updates here, I am going to close this issue.

@MarwanZeitoun if you’re still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

Can we reopen please? @google-oss-bot

We have just over 100 functions, and similarly have had a lot of issues with deploy reliability/quota issues (including exhausting the daily quota limit). We are currently using a custom deploy script to deploy them in batches of 20 functions at a time using --only - this was the only way to make continuous deployments reliable, however deploys take 30~45 minutes. (If there’s interest I can tidy up and share this script!)

I’ve updated to 11.14.0, re-run the deployment on our CI server twice (no filesystem/env changes) and I’m not seeing any skipped functions - all are re-deploying. We’re not using codebases at the moment - just a single “default” codebase.

The two things that are preventing us from migrating to codebases are:

1. We have some common modules/shared code - for managing config, firestore setup and getters/setters, firebase auth helpers, internal utils. We could split these out using yarn workspaces as mentioned above, but this adds extra developer overhead and complexity for testing/mocking/etc. I’d prefer a solution that just re-deployed all functions if a specific folder of shared code changed, or ideally operated on a per-function basis - e.g. by hashing each bundled & tree-shaken function? (the ESBuild approach mentioned above sounds great to me!)

2. Since the names of HTTP functions form the public function URL (https://us-central1-myproject.cloudfunctions.net/myFunctionName), we can’t rename these functions since the URLs are used by external services and in native client apps. (We now give new HTTP functions nice URLs using firebase hosting rewrites, but that doesn’t help with existing URLs that are in use). If we had control over the function URLs or if we could maintain the current URLs (/myFunctionName rather than /mycodebase-myFunctionName) then we could get around this.

@dtran320 …

If you want to check, I’d suggest doing thee following from a single box

1. make sure you’re on > firebase-tools@11.13.0
2. run firebase deploy
3. run firebase deploy again (from the same box)

You should see:

• functions: Skipping the deploy of unchanged functions with experimental support for skipdeployingnoopfunctions
• functions[ ... ] Skipped (No changes detected)

I only want to deploy cloud functions so I need to run firebase deploy --only functions, and this way the skipdeployingnoopfunctions doesn’t run/work.

We’ve currently launched (but not well documented or added wizzard support for) codebases. With codebases you can have multiple directories of Cloud Functions. Simply create a firebase.json fragment that looks a bit like:

"functions" : [
  {
    "source": "functions",
    "codebase": "main"
  }, {
    "source": "otherfunctions",
    "codebase": "other"
  }
]

With this, you can now call firebase deploy functions:main or firebase deploy functions:other. Next, we’re working on recognizing that a codebase hasn’t been modified since last deployment and we’ll skip those functions altogether.

Unfortunately I am also experiencing these issue often (if not each time) trying to deploy all my GCF at once

⚠ functions: got "Quota Exceeded" error while trying to update *********** Waiting to retry...

I know it’s best to deploy only the group of functions that have been updated and leave the other ones untouched, but sometimes when updating environment configuration it is required to deploy all cloud function to allow the new configuration to propagate properly.

Please advise on this issue, it would be great to find a better workaround to allow better integration with CI too.

Thanks!

Since there haven’t been any recent updates here, I am going to close this issue.

@MarwanZeitoun if you’re still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

I was chatting to some friends who use AWS - looks like lambda uses the esbuild approach: https://github.com/aws/aws-cdk/tree/main/packages/%40aws-cdk/aws-lambda-nodejs Could be useful for reference ☺️

We have about 150 functions.

% firebase -V
11.14.0

During the first deploy, 33 functions failed. Second attempt: seen some Skipped (No changes detected), but only a handful (10). Mostly firebase was trying to deploy the same function again even after a successful deployment.

These errors are appearing much more often than they used to: Failed to configure trigger for event-type:providers/cloud.firestore/eventTypes/document.write resource:projects/ourproject/databases/(default)/documents/payments/{id} service:firestore.googleapis.com.

If I try to deploy again it usually works. So it is not a configuration issue.

Is it because we are using 1gen functions? Using the workaround above not really practical if 20/100 functions failed to deploy.

Please try to fix the root cause and please perhaps add a --force option so all functions can be re-deployed even if now changes are detected.

@dtran320

Looking at the changelog for firebase-functions 11.13.0, could you help clarify what the change from #5032’s flipping of the skip no-op deploys flag to true does?

That enables the no-op deploy detection by default.

Does that ONLY apply to diff-ing the hashes for entire codebases and only applying a skip no-op deploy for a set of codebase functions, or is it meant to hash each individual function and skip the no-op function deploy for each individual function?

Applying skip deployment of no-op deploys is applied to a set of functions within a codebase.

We updated our fork of the firebase-action for Github to use firebase-tools 11.13.0 but I believe I’m still seeing it deploy every function even though they should all be no-ops.

That shouldn’t be the case… diffing codebases occur by checking:

1. the codebase’s folder
2. environment variables
3. secret versions

If you want to check, I’d suggest doing thee following from a single box

1. make sure you’re on > firebase-tools@11.13.0
2. run firebase deploy
3. run firebase deploy again (from the same box)

You should see:

• functions: Skipping the deploy of unchanged functions with experimental support for skipdeployingnoopfunctions
• functions[ ... ] Skipped (No changes detected)

This is encouraging to hear; thank you for the update!

Experiencing same issue while deploying 62 functions.

⚠  functions: got "Quota Exceeded" error while trying to update projects/<redacted>/locations/us-central1/functions/<redacted>. Waiting to retry...

2 functions fails to deploy, thus I assume there should be somewhere 60 function deployment limit?

As @42ae mentioned previously, sometimes bulk deployment is needed in order for all functions to be updated. Looking forward hearing for solutions to this problem.

My error was solved by upgrading my billing or plan to Blaze before I could deploy functions.

HOW do you solve this issue? Tried to deploy one by one and still getting errors.

To try redeploying those functions, run: firebase deploy --only “functions:sendMessageNotification,functions:groupAdminChanged,functions:deleteMessageForGroup,functions:setStatusCount,functions:groupInfoChanged,functions:deleteMessageForBroadcast,functions:unsubscribeUserFromBroadcast,functions:unsubscribeUserFromTopicOnDelete,functions:saveUidOnLogin,functions:indexNewGroupCall,functions:sendMessagesForGroups,functions:addUserToGroup,functions:participantRemoved,functions:participantAdded,functions:groupEvents,functions:deleteMessage,functions:getVirgilJwt,functions:sendMessageToBroadcast,functions:indexPKToken,functions:sendNewCallNotification,functions:subscribeToBroadcast,functions:indexNewCall,functions:sendUnDeliveredNotifications,functions:deviceIdChanged,functions:resubscribeUserToBroadcasts,functions:getTime”

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly. mac@bogon functions % firebase deploy --only “functions:getTime”

=== Deploying to ‘ichat-daac0’…

i deploying functions i functions: ensuring required API cloudfunctions.googleapis.com is enabled… i functions: ensuring required API cloudbuild.googleapis.com is enabled… ✔ functions: required API cloudbuild.googleapis.com is enabled ✔ functions: required API cloudfunctions.googleapis.com is enabled i functions: preparing functions directory for uploading… i functions: packaged functions (73.93 KB) for uploading ✔ functions: functions folder uploaded successfully i functions: updating Node.js 10 function getTime(us-central1)…

Functions deploy had errors with the following functions: getTime(us-central1)

To try redeploying those functions, run: firebase deploy --only “functions:getTime”

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly. mac@bogon functions % functions % firebase deploy --only “functions:resubscribeUserToBroadcasts” mac@bogon functions % firebase deploy --only “functions:resubscribeUserToBroadcasts”

=== Deploying to ‘ichat-daac0’…

i deploying functions i functions: ensuring required API cloudfunctions.googleapis.com is enabled… i functions: ensuring required API cloudbuild.googleapis.com is enabled… ✔ functions: required API cloudfunctions.googleapis.com is enabled ✔ functions: required API cloudbuild.googleapis.com is enabled i functions: preparing functions directory for uploading… i functions: packaged functions (73.93 KB) for uploading ✔ functions: functions folder uploaded successfully i functions: updating Node.js 10 function resubscribeUserToBroadcasts(us-central1)…

Functions deploy had errors with the following functions: resubscribeUserToBroadcasts(us-central1)

To try redeploying those functions, run: firebase deploy --only “functions:resubscribeUserToBroadcasts”

To continue deploying other features (such as database), run: firebase deploy --except functions

Error: Functions did not deploy properly.

Having trouble? Try firebase [command] --help

@abeisgoat I’m also facing a quota issue, although I’m trying to deploy just 20 functions. Can you confirm if it’s related or it’s another issue? The output that I have in Functions dashboard is:

{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":8,"message":"Build failed: Quota exceeded for quota metric 'Build Create requests' and limit 'Build Create requests per minute' of service 'cloudbuild.googleapis.com' for consumer 'project_number:<NUMBER>'."},"authenticationInfo":{"principalEmail":"<EMAIL>"},"serviceName":"cloudfunctions.googleapis.com","methodName":"google.cloud.functions.v1.CloudFunctionsService.UpdateFunction","resourceName":"projects/ticker-dev-4d59e/locations/southamerica-east1/functions/<FUNCTION_NAME>"} 

Deploying the functions individually works as expected, but trying to deploy all of them at the same time outputs me the error above.

I only want to deploy cloud functions so I need to run firebase deploy --only functions, and this way the skipdeployingnoopfunctions doesn’t run/work.

@pedrofsantoscom This was not working for me either, after digging through the code I found that the issue was with runtime configuration producing a new hash every time. I have a PR fix for this open.

• skipdeployingnoopfunctions

How do I disable this new default behavior?

There isn’t a feature to disable the behavior. (but that could be a worthwhile feature request to gather public opinion.)

That said, if you invoke deploy on a function with the --only flag, you force deployment.

Example:

firebase deploy --only functions:makeUppercase

^ This will forcefully deploy the makeUppercase function.

More info on the --only deploy

Hey @MarwanZeitoun. We need more information to resolve this issue but there hasn’t been an update in 7 weekdays. I’m marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

Thanks so much for all the advice @taeold.

Unfortunately for us, refactoring to move to codebases isn’t feasible at the moment.

Looking at the changelog for firebase-functions 11.13.0, could you help clarify what the change from #5032’s flipping of the skip no-op deploys flag to true does? Does that ONLY apply to diff-ing the hashes for entire codebases and only applying a skip no-op deploy for a set of codebase functions, or is it meant to hash each individual function and skip the no-op function deploy for each individual function? We updated our fork of the firebase-action for Github to use firebase-tools 11.13.0 but I believe I’m still seeing it deploy every function even though they should all be no-ops. Thanks so much!

@s11richard @hefgi Thanks for reaching out!

While I can see improvements in trying to improve experience in deploying large amount of functions with a single deploy command, it’s always going to be hard because Google Cloud Platform’s quota is going to be dynamic and different for every user. We also noticed that it often changes over time and can be different for each project for the same user.

Our current function deploy impl. is an attempt to navigate the quota as best as we could, and while I’m pretty sure that it’s an improvement over our past impl., it’s obvious from this issue that we still have much to go to cover everyone’s use cases.

Lately, we’ve been experimenting from another direction:

Is there a way to reduce the number of functions that’s deployed on each deployment?

Say make a code change that affects runtime behavior of a subset of your functions. Do we really need to deploy all of the functions?

@hefgi’s suggestion of breaking your functions into codebases is part of an answer. But it still means you need to manually pick which codebases to deploy on each run.

@TheIronDev recently worked on a new feature to automatically detect and skip deployment of functions in a codebase without any changes.

I’d strongly recommend you give codebases and “no-op deploy” a try. In the coming weeks, we’ll try to publish more public guidance on how we think we can improve reliability of deploys for projects with many many functions.

@gustavopch I think somewhere between the 2 extremes (1 codebase 100 functions vs 100 codebase 1 function) would help. For example, you might group function in a codebase by trigger type. Or by their business domain. In the ideal case, when you are apply a bug fix or a new feature, it would affect a single codebase. But I suspect that our reality is a bit messier than that.

You can give “skip deploying a codebase that’s doesn’t require one” right now by using the latest Firebase CLI and enabling the priview:

firebase --open-sesame skipdeployingnoopfunctions

We are making some last pushes to get this feature cleaned up, an when its ready you see it be mentioned in our documentations. Would love to hear from advanced users like yourselves what you think.

@inlined is it possible there is a regression and https://github.com/firebase/firebase-tools/issues/2606 / https://github.com/firebase/firebase-tools/pull/3246 no longer works? Is it possible to restore ‘retry on quota’ functionality?

I am frequently experiencing this issue as well while deploying over 100 functions. We deploy in groups and have created some parsing logic for our CI system, however it would be very convenient to have these auto-retry if we pass the deployment quota, or at least spit out a command after failure to deploy the rest of the functions (currently it just lists them).

The current text says “waiting to retry” while we exceed the quota, so that was the behavior I was expecting, however it is not what ended up happening.

This is also happening for us pretty consistently, 3-5 will fail with quota exceeded when deploying 60 functions. We’re using firebase-tools 9.23.0.