firebase-tools: Deploying functions extremely slow

This issue is a real pain in the bum. I think it should be given high priority 😕

@laurenzlong As a newbie to Firebase at first glance I thought I was messing up something, because the deploy always stopped at the functions: preparing functions directory for uploading… part. Giving some extra info (like: it takes 3-5 minutes to complete) would be really nice, until the problem is solved.

As a newbie, I spent many times canceling on this thinking that I was doing something wrong.

Thanks for filing! We know that slow deployment is a major pain point for the functions experience, and it is something that we are working to resolve through various strategies.

@McStuffins If your car takes five minutes to turn on, is it an error or simply an issue with start time?

Day 19. Firebase still deploying. * grabs popcorn *

@SrBrahma sure, but the issue is still present even if you only deploy a single function (e.g., single function deploy can still fail due to timeout).

It shouldn’t have to fall on the user to manage batching function deployment, and in many cases function updates are needed atomically, therefore can’t be split/batched.

Appreciate the suggestions for a workaround, but really what we need here is an SLA from Google and their adherence to it.

@RenFontes I just filed Case 00075974: Deploying firebase functions is so slow as to be unusable with Firebase support. I will update this message with whatever they come back with.

Copying your functions directory, including node modules, to a tmp dir.

Our next release will address this and no longer need to copy before deploying.

On Sun, Jan 7, 2018, 5:05 PM hmexx notifications@github.com wrote:

I’m really not sure what firebase could be doing that takes MINUTES to deploy a 5 line, 1kb function, on a six 4GHz core machine, sitting on a 1Gbps fiber connection.

I know it sounds like I’m taking the piss, but I’m genuinely curious what is going on during “preparing directory for upload”. Anyone actually know?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/536#issuecomment-355868154, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAD_nUptJWFuYGvEXI0MwmQR-bG9_MKks5tIWm1gaJpZM4QdF3g .

I kept track of this issue from the beginning, but it’s never been a problem for me since I have a CD set up and it does all the work for me. I also never deploy functions just to test if they work. So basically it didn’t matter to me.

Until today, when I came across an unexpected limitation: I cannot deploy my functions anymore, cause deployment to production exceeds the daily quota (12,000 seconds). I have ~55 functions with various triggers (pubsub, firestore, https). Is it too much to handle?

Now I have to deploy my application for two days 🤣 🍭 👍 🥇 ⚰️ 🎉 🌮 🌵 💃 😈

@laurenzlong thank so much for responding and thanks for your hard work on Firebase!

Are there any active plans to improve deploy time? I ask because although we do like Firebase, we are actually thinking of moving away from it due to the slow deploy times. We have about 47 cloud functions and it regularly takes 3-6 minutes for our deploys. Compare that with git push heroku master which would take about 20-30 seconds to deploy the same volume of code to Heroku. And that code would then be instantly running on Heroku. With Firebase we have to wait another 20-30s (random) amount of time after deployment before the deployed functions are actually running (during this time, a mix of new and old functions run).

So compare the deployement experience:

*** Heroku:

git push heroku master

20-30s later ...

All newly-deployed functions are now running in a consistent/atomic way

*** Firebase:

firebase deploy --only functions

180-360s later ...

Functions are deployed but only some of the new ones are running, some old ones are still running

20-30s later ... 

All new functions running

It’s also difficult to do atomic deploys with Firebase because after deployment, there is period of time during which some of the new functions, call them v_n+1 are running alongside some of the old functions v_n. So if you do a major update, you have a mix of new and old functions running possibly using different data formats or algorithms. This is much less safe than a Heroku deploy where either all the new functions deploy and only new functions are running or none deploy.

Further, sometimes a few of our 47 functions will fail to deploy with errors like

⚠  functions[retrieveFavorites(us-central1)]: Deployment error.
Server Error. getaddrinfo ENOTFOUND cloudfunctions.googleapis.com cloudfunctions.googleapis.com:443

We’re deploying on highly reliable symmetric Gigabit internet, so the issue is not our network.

So think of Heroku deploys as a like an atomic DB transaction (all or nothing) whereas Firebase deploys are more like eventually consistent with partial failures … This makes dev ops a lot more difficult, especially if you’re pushing a bug fix in the middle of night in response to a page.

The be blunt, the deploy experience on Firebase is objectively slower and less reliable than on Heroku or AWS … Don’t get me wrong, we like Firebase and we really do appreciate all of your hard work. I don’t mean this to sound like an attack, it’s nothing personal, but we need Firebase to do better here or we’re thinking of going elsewhere because deployments are just too painful.

Thanks again for your hard work on Firebase. We appreciate your help 😃.

@laurenzlong-

However, these are infrastructure changes that will take quarters, so we really appreciate your patiences in the meanwhile.

Any infrastructure-side updates since your last note a few quarters ago? Until these changes have been made, is it possible to re-open this issue? Closed does not seem to be the appropriate state unless tracking has moved to a different/related issue.

Firebase functions are still super slow after everything has been built/uploaded and the functions are being created/updated on the server side of things. As others have said, this is a major development velocity damper.

firebase --version
7.1.0

Many thanks for considering!

Just poking my head in to remind people that this is still a pain point ✌️

Happy New Year to this issue that is still Closed and still not fixed! 🎉

Over here we’ve also been grouping related functionality into a function, so it ends up looking like "one function per feature area” rather than “one function per individual granule of functionality”.

The performance characteristics of deployment are strongly driving the architecture of what goes into one function!

@horacehylee @merlinnot Thanks for the 2 cents. Please see https://github.com/firebase/firebase-tools/pull/578, the next release of the CLI will no longer copy the functions source folder period.

my console is still deploying still 2017, I’m still positive for a fix, i will keep it open

@chriscurnow it was closed because the real cause is apparently is due to the (closed-source) server side not the CLI tools, even though it manifests to end users as a CLI issue (per @samtstern in https://github.com/firebase/firebase-tools/issues/536#issuecomment-572830647).

Unfortunately there’s no good public tracker for the back end so we get updates here 😦

Still, this quota limit seems really low, since the official doc lists “80 per 100 seconds” for “Calls to deploy or delete functions via the Cloud Functions API”. I’m assuming Google support could not raise this limit even further?

@dinvlad I was inquiring to both sales and support about getting that API limit increased, it’s explicitly disabled / greyed out in the GCP console quotas section. Eventually was able to get a Google engineer on hangouts who informed me “That quota does not change”… so I guess not.

Our firebase app has 60 cloud functions running in it now. At first we started having a lot of deployment failures due to quotas so we then broke it into batches as the docs suggest. It consistently deploys now, but each batch takes ~3 minutes to deploy as part of GitHub actions CI on the default runners. The batches are about 6 each so 10 batches total making our function deployment time around 30 minutes.

The functions themselves are very small bits of code with minimal dependencies. Not sure what else we can do to speed up the pipeline.

For me this still takes minutes.

I have firebase-tools@3.17.4 and firebase-functions@0.8.1. What can I do to help further debug this issue?

The release should happen this week, and you can expect the “Preparing functions directory for deploy” step to go substantially faster (I don’t have exact numbers). Other parts of the deploy will remain unchanged.

On Mon, Jan 15, 2018 at 12:42 PM hmexx notifications@github.com wrote:

Yeah.

When’s the next release slating for, and what % improvement in deployment times should we expect?

Just to know whether to keep going down this path.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/536#issuecomment-357784792, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAD_vaj-WngS9dMj7j5Q2AesM0tNMvVks5tK7hBgaJpZM4QdF3g .

@laurenzlong I would have to configure my CI to automatically detect changes in each function between deployments (including resolving dependencies). And what if I update packages like firebase-functions, firebase-admin, lodash etc., which I use in every single function?

FWIW I believe these accept globs so you might be able to do:

{
  "exclude": ["!{lib,package.json,package-lock.json}"]
}

This issue was closed because there’s nothing more we can do on the CLI side to improve deploy times. For local development we’re continuing to invest in the emulator, we hope that people will have to run firebase deploy only when they’re deploying to prod and not as part of their dev lifecycle.

Backend changes are still in progress. As you can tell, there have been unexpected delays.

This was released!

The problem is kind of obvious. The init command created a ‘node modules’ folder that is huge. On my machine it is: 22,903 Files, 2,782 Folders. The code copies all of that to a temp folder.

Here is what I did:

1. Using log print statements, I identified the slow line of code. It is here: https://github.com/firebase/firebase-tools/blob/master/lib/prepareFunctionsUpload.js#L26

Line 26 in prepareFunctionsUpload.js: fs.copySync(options.config.path(options.config.get(‘functions.source’)), tmpdir.name);

1. I printed out the parameters for that file copy call. On my machine it is: C:\Users\thoma\StudioProjects\LSystemAndroid\firestore\functions -> C:\Users\thoma\AppData\Local\Temp\fbfn_752624vejX0cv3GEJI

The functions folder was created by the CLI command init. It looks like this:

• node_modules (enormous size)
• .gitignore
• index.js
• package.json
• pacakge-lock.json

It seems to me that the CLI command init should NOT have created that node_modules folder. It’s 165MB large. That seems unreasonable to add to every project.

@RenFontes I just filed Case 00075974: Deploying firebase functions is so slow as to be unusable with Firebase support. I will update this message with whatever they come back with.

Did they get back to you with any solutions? @RenFontes

Not really @Scino. This is what I received from Firebase support, and I didn’t follow up further. Nobody on the Firebase/Google team seems to appreciate that every now and then we actually need to push functions to production rather than just play with them in an emulator. Single function uploads often fail, even if they have been tested and validated locally via the emulator. Good luck trying to get multiple updates to push in any sort of atomic way.

I’m Joel, and I’ll be happy to assist you here.

Cloud Functions deployments do take some time, and the team is working to reduce the time it takes. However, the next steps might help in case the delay exceeds the usual timing.

Please try the following: Start with a clean slate. (new directory) Upgrade to the latest Firebase packages: firebase i -g firebase-tools@latest firebase i firebase-functions@latest firebase i firebase-admin@latest Try with a “hello world” function with no third party dependencies. Check that the network is stable. Factors that can affect are internet speed, network glitches, compilation (in case of typescript and/or other tools) Deploy. You can deploy only the Functions components with: firebase deploy --only functions

Even deploying a single function like: firebase deploy --only functions:my-function

Please use the --debug flag if necessary to troubleshoot any issues (firebase deploy --debug --only functions:my-function)

I hope you find this information useful.

Cheers, Joel

I’m writing this comment in the midst of a major outage that I’m in charge of fixing. “But @lilred”, you might say, “why are you replying to closed GitHub issues instead of attending to your major outage?”

Because every time I attempt a fix and re-deploy a single Firebase Function, I have to wait and twiddle my thumbs for five minutes and a half before deployment completes and I can see if my fix worked. And maybe I need to make two or three attempts. At this point more than half of our backend incident response time is waiting on Cloud Functions deploys.

@royappa @SrBrahma thank you for sharing your emulators solutions, one major goal of the emulators is to reduce how often developers have to run firebase deploy! For my personal projects the emulators have totally changed my development experience, I never deploy just to test something now.

Deploying the initial helloWorld sample function takes 1 min 40 sec with the above “.ignore” rule (Jan. 2021) so I think it’s safe to say this is not a priority for Google to improve.

Just in case this helps any other beginners: run the Firebase emulator, then change “tsc” to “tsc -w” (if using typescript) in package.json, and finally run “npm run build” in another window. With this, the emulator reloads your function changes right away. This makes local development a lot faster than waiting 2 minutes to test changes on Google’s servers.

I have made this start script in package.json some time ago, that is incredibly helpful and useful to quickly start the emulator and set tsc watch with a single command:

"start": "tsc --incremental && concurrently 'tsc -w' 'firebase emulators:start --only functions,database,auth --export-on-exit=./localSave --import=./localSave'",

Install concurrently with npm i -D concurrently, it runs multiple commands in parallel and in same terminal. I don’t remember now why I added the --incremental flag there in tsc, but it certainly fixed something. Export and import are optional fields to save the state of the emulator to use on other sessions.

I also have the startClean script that removes previous state data:

"startClean": "rm -rf ./functions/localSave && npm run start",

Edit: I think I remembered about the incremental: first we build project with tsc, so when the emulator is started, it’s using the latest code. Using incremental prevents that tsc -w rebuilds the project again (taking longer), already done by the first tsc.

Still waiting for Google to improve Google Functions deploying time right ? Going from PWA to SSR forced me to wait 7 min more for every deploy 😞

Perhaps an admin or OP can rename this issue to “Preparing functions directory for deploy extremely slow” since it seems to have been closed based solely on a fix to that stage of the overall deploy process.

We can then open a new “Deploying functions extremely slow” issue since … deploying functions is still extremely slow 😄

@repentsinner the backend project to improve build/deploy times is still ongoing (and going well!) but it’s still not released.

Have you tried using the emulator (through firebase emulators:start) to speed up your development process? If so we’d love to get your feedback!

Last couple days deploying even hosting takes multiple minutes, when it took like 10-15 seconds couple weeks ago. Deploying functions times out as well.

And the Firebase status page is green https://status.firebase.google.com/ .

Does anyone know why this happens?

Using firebase-tools 7.0.0

$ firebase deploy --only=hosting --token xxx

=== Deploying to 'xxx'...

i  deploying hosting
i  hosting[xxx]: beginning deploy...
i  hosting[xxx]: found 1959 files in public
✔  hosting[xxx]: file upload complete
i  hosting[xxx]: finalizing version...
✔  hosting[xxx]: version finalized
i  hosting[xxx]: releasing new version...
✔  hosting[xxx]: release complete
✔  hosting: Finished running postdeploy script.

✔  Deploy complete!

Project Console: https://console.firebase.google.com/project/xxx/overview
Hosting URL: https://xxx.firebaseapp.com
✨  Done in 181.75s

Yes you’re right, I was mistaken, node_modules does get copied over. I think it is a valid idea to not copy over node_modules to the temp directory. What complicates this a bit is the fact that the CLI writes a “.runtimeconfig.json” to the temp folder prior to trigger parsing, and this file gets uploaded with the rest of the functions source code, and we didn’t want to write this file into the actual source code directory. So there is probably a good solution that both improves deployment speed and ensures there aren’t unintended side effects, but I’d have to play around with a bit. You can also feel free to make a pull request.

@RenFontes I just filed Case 00075974: Deploying firebase functions is so slow as to be unusable with Firebase support. I will update this message with whatever they come back with.

Did they get back to you with any solutions? @RenFontes

We’re facing the same issue, as even single functions deployment is so slow that creating new functions inevitably fails, unless we force a longer timeoutSeconds in function’s runtimeOpts before deploying. That, for instance, is an ugly workaround, but not a real solution.

Also it’d be nice to have more useful error logs for these timeout-related deployment errors, rather than just a Error: Functions did not deploy properly. message. We wasted too much time trying to figure out what could cause the issue by attempts.

Deploying the initial helloWorld sample function takes 1 min 40 sec with the above “.ignore” rule (Jan. 2021) so I think it’s safe to say this is not a priority for Google to improve.

Just in case this helps any other beginners: run the Firebase emulator, then change “tsc” to “tsc -w” (if using typescript) in package.json, and finally run “npm run build” in another window. With this, the emulator reloads your function changes right away. This makes local development a lot faster than waiting 2 minutes to test changes on Google’s servers.

That’s really sad to hear. Another approach is to bundle multiple entrypoints/routes into a single function. This is not the best from the separation of concerns/size/security obviously, but this is what we did with all of our API endpoints (single entrypoint that then uses an internal router), and haven’t experienced many quota limits since (but they do happen occasionally).

Still, this quota limit seems really low, since the official doc lists “80 per 100 seconds” for “Calls to deploy or delete functions via the Cloud Functions API”. I’m assuming Google support could not raise this limit even further?

painful deploy time 😦

Just started cloud functions, and the deploy time is painful.

That’s an extract from my firebase.json:

  "functions": {
    "ignore": [
      "__mocks__",
      ".cache",
      ".commitlintrc.yaml",
      ".dependabot",
      ".editorconfig",
      ".eslintrc.yaml",
      ".firebase",
      ".firebaserc",
      ".git",
      ".gitattributes",
      ".github",
      ".gitignore",
      ".lintstagedrc.js",
      ".nvmrc",
      ".prettierignore",
      ".prettierrc.yaml",
      ".vscode",
      "CHANGELOG.md",
      "cloudbuild.yaml",
      "codecov.yml",
      "CONTRIBUTING.md",
      "coverage",
      "cSpell.json",
      "decisions",
      "firebase.json",
      "firestore.indexes.json",
      "jest.config.js",
      "node_modules",
      "README.md",
      "rfcs",
      "scripts",
      "src",
      "test",
      "tsconfig.json",
      "tsconfig.production.json"
    ],
    "source": "."
  },

I’d gladly replace it with:

{
  "include": ["lib", "package.json", "package-lock.json"]
}

I dropped functions completely as the dev workflow is beyond painful.

The emulator doesn’t support auth context so its 100% completely useless IMO.

+1 That deploying a function is extremely slow. I followed the cron job tutorial for creating the hourly-tick function, my very first tone, and it takes like a minute to deploy every single time.

Functions have always been deployed in parallel. However, please note that deploying 10 functions in parallel is still slower than deploying a single function.

Even deploying from google cloud build it takes 2 to 3 minutes, versus 10 to 20 seconds in aws lambda. I just want to ask does the cli zipping the whole function folder before uploading?

There has not been a regression in deployment strategy in recent firebase-tools versions (the node_modules folder is not copied), however deployment time does also vary machine to machine and from time to time. Initial deploy of a function is also slower than subsequent updates.

npm install -g npm@latest gave me slightly different input, and removing firebase-cli seemed to change something too.

update:

okay, after running npm install -g firebase-tools it finally worked. Kudos to https://stackoverflow.com/questions/48531993/firebase-config-variables-are-not-available-error-with-deploying-functions#comment84098334_48531993

@mbleigh

Ah right. that explains it. It’s kind of comedic seeing a workstation go crazy for a minute or two, only to have the next line print out “packaged functions (37.55kb !!!) successfully for uploading” lol

Look forward to the next release. Thx for responding.

H.

For everyone interested: https://issuetracker.google.com/issues/71385193

Sometimes, when I am deploying, it is extremely slow, and then there is a warning in the terminal that says “Error in the build environment”

I put a debug print line before line 26 and after. That line took minutes.

Then, I added a filter to print out which files were being copied. It included all the node_modules files.

Then, I changed the filter to exclude the node_module files. The deployment progressed fast now. However a little later, it seemed that the script was trying to evaluate the correctness of the cloud functions. That code failed because it was missing the library dependencies.

The source code line that you point to seems to be a later step. It seems like the files (minus the node_modules) are archived into a zip file before uploading to the cloud. That line doesn’t seem to run slow on my machine.

@thomasfischersm

That’s actually not the issue. The node_modules is not copied over to the temporary folder, as you can see from this line here: https://github.com/firebase/firebase-tools/blob/master/lib/prepareFunctionsUpload.js#L76

It is necessary for firebase init to install all the node modules locally inside of the functions source folder, otherwise locally serving the functions will not work, and trigger extraction will not work (this is how the CLI understands which events trigger each function so it can deploy them properly).