firebase-functions: Functions deployment fails when firebase-functions has been hoisted by a monorepo manager like yarn/workspaces or lerna

This is an issue I’m currently facing.

Would this be a feature request that can easily be added via a PR, or are there deeper changes that would need to be made? If a PR is feasible then I wouldn’t mind making an addition if pointed in the right direction.

@rtm I’m talking about when you have a mono repo, where one project/package is the cloud functions project and let’s say we have utilities projects. Now, I want to use the utilities project in my cloud functions project. Yarn workspaces would hoist that into the node_modules folder of the cloud functions project. However, during deployment, this won’t work, because could functions look inside the package.json for dependencies, and it won’t find my utilities project on the NPM registry. My question: does nohoist solve that problem somehow?

@laurenzlong Thanks again for your pointers. Yes, I know how to use the functions property in firebase.json, and the main property in package.json, and in fact am already using both of those. Neither helps with the basic problem that if some monorepo manager (be it yarn/workspaces, or lerna) has hoisted packages from my-repo to one level higher, the deploy fails.

As I said, there may be magic things that the logic for deploying functions is doing that make it hard to deal with this scenario. At the same time, monorepos are a “thing” now and I predict that more and more people will be trying to build their functions directory as a sub-repo in a monorepo. For example, I have a monorepo with sub-repos for my mobile app, my webapp, and shared database logic, so it makes perfect sense to have cloud functions as another sub-repo, and I doubt if I’m alone in this kind of thinking. So at some point it could become more important to see if this could be made to work.

The following note from the Lerna site on hoisting might be instructive:

Unfortunately, some tooling does not follow the module resolution spec closely, and instead assumes or requires that dependencies are present specifically in the local node_modules directory. To work around this, it is possible to symlink packages from their hoisted top-level location, to individual package node_modules directory. Lerna does not yet do this automatically, and it is recommended instead to work with tool maintainers to migrate to more compatible patterns.

Lerna makes hoisting optional with the --hoist flag, but turning it off sort of defeats the point. I haven’t tried that with Firebase functions deployment but suspect it would work fine. Yarn, on the other hand, provides no way to control hoisting when workspaces are involved–everything that can be is always hoisted.

I tried using the symlink technique mentioned on the lerna site, symlinking functions/node_modules/firebase-functions to ../../node_modules/firebase-functions. However, this does not work either, perhaps because of the behavior of the deployment logic in following symlinks–the same problem that angular-cli solved with the addition of the --preserve-symlinks options. In any case, even if they worked symlinks would not be a very robust solution to the problem, since we’d need additional logic here and there to create them every time we did the equivalent of an npm install.

@laurenzlong Many thanks for your response. I’ve had functions deploying and working just fine for some time now, so I know how to do it. The issue is getting it to work with my mono-repo setup. I don’t think it’s such a terribly unique use case; many people are starting to use mono-repos, especially with the new yarn “workspaces” feature. It’s natural to want to structure my Firebase functions as one (sub)repo within my mono-repo.

The way the yarn mono-repo support works, as you probably know, is that dependencies listed in the package.json file of subrepos are (usually) “hoisted” and placed under the node-modules folder at the very top, the better to share them. That works fine in any scenario involving standard node module resolution semantics.

In the case of firebase functions, I obviously don’t know all the internal issues or constraints that are forcing all dependencies to be found directly within the functions/node_modules directory. (I do know that that seems to sort of violate the well-understood semantics of where npm packages are located and how they’re searched for.) It would be nice if you could figure out some way to make functions work without this restriction. So I guess you could consider this post a feature request. 😃

FYI, I tried webpack-ing the code including all the npm packages such as firebase-functions, while leaving dependencies on built-in node packages like fs external. That resulted in a 7MB index.js, but that’s OK. However, when I tried to deploy that, I got a different error

Error: Firebase config variables are not available. Please use the latest version of the Firebase CLI to deploy this function. `

The code seems to be looking for a JSON file called ../../../runtimeconfig.js, or an environment variable called FIREBASE_CONFIG_FILE or something pointing to such a file, which apparently needs to contain a firebase key, but I can’t find any information about such a file, what it’s supposed to contain, or why standard deployments work without such a file.

At that point I ran out of gas and gave up.

yarn now supports this scenario if you use workspaces:

"workspaces": {
  "packages": ["cloud-functions"],
  "nohoist": [
    "**/firebase-admin",
    "**/firebase-admin/**",
    "**/firebase-functions",
    "**/firebase-functions/**"
    ]
}

That’s what I do now and it works fine.

If I understand the deployment method of cloud function correctly, at least via the firebase-cli, no node_modules folder will ever be send to the cloud functions server. Only your code + the package.json is send. Cloud functions will then run npm install. The problem is, that any hoised or linked yarn/npm link package is not found my the cloud function.

Here is what I do as a work around. It’s ugly but it works. Let’s say i have a utilities project, and a functions project. I will use yarn/npm link to connect the functions and utilities with each other. That works fine during development. When deploying, I do the following. First, I have a script that makes a npm package out of the utilities folder (npm pack does the trick). Second, the script copies the packages into a subfolder in my functions project. Let’s call it cdn. This folder will get deployed with the rest of the code. Finally, in the package.json of my functions project I add the following dependency "utilities": "file:src/cdn/utilities-1.0.0.tgz",. With this setup, all the code is sent to the cloud function. And running npm install will install my utilities project.

This is a really annoying issue. I wanted to be able to share code between my packages, which lerna takes care of through symlinks. My solution is to have pre and post deploy scripts that copy the functions package into a new temporary directory, and use that as the thing that’s deployed. It goes into the new temp dir’s package json and modifies the version field of the shared package such that it is now of the format “file:./temp_cloned_packages/@mygroup/sharedcodepkg”

This is a janky solution, would really like for the firebase cli to support symlinks.

Yarn solution

In your path/to/functions/package.json add the following

{
  // ...
  "workspaces": {
    "nohoist": [
      "**"
    ]
  }
}

It basically says, don’t hoist anything in this package, so will always retain the node_modules directory.

It does marginally slow down installations but I haven’t noticed anything drastic.

@laurenzlong Appreciate you taking care of this.

It had already occurred to me that this was a cli issue. Thanks for moving it over there.

I’ll defer to your superior knowledge, but I can’t see why there would be implications for non-function deployment.

@laurenzlong Yes, I have a mono-repo, where the sub-repos include one for a Cordova app and one for a desktop/browser webapp (which we call the “console”), as well as one that provides shared Angular components, one that provides standard data types and utilities for standard objects used in the app and console, and then one with some basic JS utilities. And then finally I have the sub-repo that contains cloud functions. All the firebase-related sub-repos (app, console, and cloud) consume (have as dependencies) the common subrepo of types and utilities. Handling these cross-dependencies in a seamless way is one of the main motivations of using a mono-repo. So it looks like this:

root
  app      <-- depends on common
  console <-- depends on common
  cloud   <-- depends on common
  common
  utils

I treat the root as the firebase home, which contains .firebaserc, firebase.json and so on. So for instance the hosting key in firebase.json would point to console/dist, and the functions key to cloud. I am now thinking about changing over to a separate sub-repo for firebase config to allow me to work more easily with multiple firebase configurations, but that’s a different issue.

The cloud directory has a src sub-directory containing the souces, and the relevant build rule (from the root) is something like

tsc --project cloud/src --outDir cloud/dist

The deploy command, from the root, is just

firebase deploy --only functions

and as mentioned above this picks up the functions key from firebase.json which has the value of cloud which contains a package.json with a main field of dist/index.js which is created by the tsc command shown above.

This has all worked fine in various permutations I’ve tried as I went along. What does NOT work fine is when the firebase-functions and firebase-admin packages, which are called for as dependencies in cloud/package.json, are HOISTED by yarn or lerna to $ROOT/node_modules. This results in the deploy error mentioned earlier in this thread.

My current work-around is just to remove cloud from the mono-repo structure. However, this is less than ideal, because this forces me to patch its dependencies on other sub-repos using things like npm link, the ability to avoid which is one of the main selling points of the mono-repo movement.

Hey @rtm, if you want to customize the behavior of deployment, here are a couple of options.

• modify the source code folder for functions in firebase.json

"functions": {
    "source": "my-repo"
}

• my-repo/package.json then becomes the package.json for your functions, you can modify it to point to another main file

"main": "functions/index.js"

.runtimeconfig.json is created by the CLI during deployment, so you are not expected to create it yourself.