firebase-functions: Functions deployment fails when firebase-functions has been hoisted by a monorepo manager like yarn/workspaces or lerna
I am using the workspaces (monorepo) feature of yarn. My functions
directory is a subrepo. That subrepo lists firebase-admin
and firebase-functions
as dependencies, but when installing packages, yarn “hoists” them, so they come under node_modules
at the top of the repo hierarchy:
my-repo
functions
package.json, with firebase-functions etc. as dependencies
node_modules
firebase-functions << yarn hoists this package here
The problem is, when I try to deploy the cloud functions, I get the message
Error: Error parsing triggers: Cannot find module 'firebase-admin'
What is apparently happening is that the deployment logic is parsing the code, and looking for firebase-admin
directly inside the functions/node_modules
directory, and not looking up the node search path as one might expect.
The only alternative I can see is to run npm install
directly inside the functions
directory, but that sort of defeats the purpose of having a mono-repo in the first place. Also, I have to redo that every time I run yarn on my monorepo since yarn would think the “local” versions of the packages under functions
are unnecessary and would hoist them again. I could remove the functions
directory from the list of workspaces, but that would lead to problems linking to other sub-repos.
Assuming I am understanding the problem correctly, what I would like to see as desired behavior is to have the Firebase functions parsing logic search for packages up the node search chain, rather than limiting its search to function/node_modules
.
I did try adding symbolic links from functions/node_modules
to $TOP/modules
. But that fails too, when the Firebase functions parsing logic tries finding dependencies called in by firebase-functions
etc.
Version info
firebase-functions: 0.8.1
firebase-tools: 3.14.0
firebase-admin: 5.8.1
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 19
- Comments: 20 (5 by maintainers)
This is an issue I’m currently facing.
Would this be a feature request that can easily be added via a PR, or are there deeper changes that would need to be made? If a PR is feasible then I wouldn’t mind making an addition if pointed in the right direction.
@rtm I’m talking about when you have a mono repo, where one project/package is the
cloud functions project
and let’s say we haveutilities projects
. Now, I want to use theutilities project
in mycloud functions project
. Yarn workspaces would hoist that into thenode_modules
folder of thecloud functions project
. However, during deployment, this won’t work, becausecould functions
look inside thepackage.json
for dependencies, and it won’t find myutilities project
on the NPM registry. My question: doesnohoist
solve that problem somehow?@laurenzlong Thanks again for your pointers. Yes, I know how to use the
functions
property infirebase.json
, and themain
property inpackage.json
, and in fact am already using both of those. Neither helps with the basic problem that if some monorepo manager (be it yarn/workspaces, or lerna) has hoisted packages frommy-repo
to one level higher, the deploy fails.As I said, there may be magic things that the logic for deploying functions is doing that make it hard to deal with this scenario. At the same time, monorepos are a “thing” now and I predict that more and more people will be trying to build their
functions
directory as a sub-repo in a monorepo. For example, I have a monorepo with sub-repos for my mobile app, my webapp, and shared database logic, so it makes perfect sense to have cloud functions as another sub-repo, and I doubt if I’m alone in this kind of thinking. So at some point it could become more important to see if this could be made to work.The following note from the Lerna site on hoisting might be instructive:
Lerna makes hoisting optional with the
--hoist
flag, but turning it off sort of defeats the point. I haven’t tried that with Firebase functions deployment but suspect it would work fine. Yarn, on the other hand, provides no way to control hoisting when workspaces are involved–everything that can be is always hoisted.I tried using the symlink technique mentioned on the lerna site, symlinking
functions/node_modules/firebase-functions
to../../node_modules/firebase-functions
. However, this does not work either, perhaps because of the behavior of the deployment logic in following symlinks–the same problem that angular-cli solved with the addition of the--preserve-symlinks
options. In any case, even if they worked symlinks would not be a very robust solution to the problem, since we’d need additional logic here and there to create them every time we did the equivalent of annpm install
.@laurenzlong Many thanks for your response. I’ve had functions deploying and working just fine for some time now, so I know how to do it. The issue is getting it to work with my mono-repo setup. I don’t think it’s such a terribly unique use case; many people are starting to use mono-repos, especially with the new yarn “workspaces” feature. It’s natural to want to structure my Firebase functions as one (sub)repo within my mono-repo.
The way the yarn mono-repo support works, as you probably know, is that dependencies listed in the
package.json
file of subrepos are (usually) “hoisted” and placed under thenode-modules
folder at the very top, the better to share them. That works fine in any scenario involving standard node module resolution semantics.In the case of firebase functions, I obviously don’t know all the internal issues or constraints that are forcing all dependencies to be found directly within the
functions/node_modules
directory. (I do know that that seems to sort of violate the well-understood semantics of where npm packages are located and how they’re searched for.) It would be nice if you could figure out some way to make functions work without this restriction. So I guess you could consider this post a feature request. 😃FYI, I tried webpack-ing the code including all the npm packages such as
firebase-functions
, while leaving dependencies on built-in node packages likefs
external. That resulted in a 7MB index.js, but that’s OK. However, when I tried to deploy that, I got a different errorThe code seems to be looking for a JSON file called
../../../runtimeconfig.js
, or an environment variable calledFIREBASE_CONFIG_FILE
or something pointing to such a file, which apparently needs to contain afirebase
key, but I can’t find any information about such a file, what it’s supposed to contain, or why standard deployments work without such a file.At that point I ran out of gas and gave up.
yarn now supports this scenario if you use workspaces:
That’s what I do now and it works fine.
If I understand the deployment method of cloud function correctly, at least via the firebase-cli, no
node_modules
folder will ever be send to the cloud functions server. Only your code + the package.json is send. Cloud functions will then runnpm install
. The problem is, that any hoised or linkedyarn/npm link
package is not found my the cloud function.Here is what I do as a work around. It’s ugly but it works. Let’s say i have a
utilities
project, and afunctions
project. I will useyarn/npm link
to connect thefunctions
andutilities
with each other. That works fine during development. When deploying, I do the following. First, I have a script that makes a npm package out of theutilities
folder (npm pack
does the trick). Second, the script copies the packages into a subfolder in my functions project. Let’s call itcdn
. This folder will get deployed with the rest of the code. Finally, in thepackage.json
of myfunctions
project I add the following dependency"utilities": "file:src/cdn/utilities-1.0.0.tgz",
. With this setup, all the code is sent to the cloud function. And runningnpm install
will install myutilities
project.This is a really annoying issue. I wanted to be able to share code between my packages, which lerna takes care of through symlinks. My solution is to have pre and post deploy scripts that copy the functions package into a new temporary directory, and use that as the thing that’s deployed. It goes into the new temp dir’s package json and modifies the version field of the shared package such that it is now of the format “file:./temp_cloned_packages/@mygroup/sharedcodepkg”
This is a janky solution, would really like for the firebase cli to support symlinks.
Yarn solution
In your
path/to/functions/package.json
add the followingIt basically says, don’t hoist anything in this package, so will always retain the node_modules directory.
It does marginally slow down installations but I haven’t noticed anything drastic.
@laurenzlong Appreciate you taking care of this.
It had already occurred to me that this was a cli issue. Thanks for moving it over there.
I’ll defer to your superior knowledge, but I can’t see why there would be implications for non-function deployment.
@laurenzlong Yes, I have a mono-repo, where the sub-repos include one for a Cordova app and one for a desktop/browser webapp (which we call the “console”), as well as one that provides shared Angular components, one that provides standard data types and utilities for standard objects used in the app and console, and then one with some basic JS utilities. And then finally I have the sub-repo that contains cloud functions. All the firebase-related sub-repos (app, console, and cloud) consume (have as dependencies) the common subrepo of types and utilities. Handling these cross-dependencies in a seamless way is one of the main motivations of using a mono-repo. So it looks like this:
I treat the root as the firebase home, which contains
.firebaserc
,firebase.json
and so on. So for instance thehosting
key infirebase.json
would point toconsole/dist
, and thefunctions
key tocloud
. I am now thinking about changing over to a separate sub-repo for firebase config to allow me to work more easily with multiple firebase configurations, but that’s a different issue.The cloud directory has a
src
sub-directory containing the souces, and the relevant build rule (from the root) is something likeThe deploy command, from the root, is just
and as mentioned above this picks up the
functions
key fromfirebase.json
which has the value ofcloud
which contains apackage.json
with amain
field ofdist/index.js
which is created by thetsc
command shown above.This has all worked fine in various permutations I’ve tried as I went along. What does NOT work fine is when the
firebase-functions
andfirebase-admin
packages, which are called for as dependencies incloud/package.json
, are HOISTED by yarn or lerna to$ROOT/node_modules
. This results in the deploy error mentioned earlier in this thread.My current work-around is just to remove
cloud
from the mono-repo structure. However, this is less than ideal, because this forces me to patch its dependencies on other sub-repos using things likenpm link
, the ability to avoid which is one of the main selling points of the mono-repo movement.Hey @rtm, if you want to customize the behavior of deployment, here are a couple of options.
.runtimeconfig.json
is created by the CLI during deployment, so you are not expected to create it yourself.