firebase-tools: Function deployment fails with npm workspaces/nx - Error: spawn ./node_modules/.bin/firebase-functions ENOENT

Just finished reading through the issue - thanks @rahuld600 @ChromeQ @dereekb @michaelangrivera and everyone else for sharing your debugging notes.

For context here’s quick summary of how function deployment works:

Firebase CLI needs to know the functions are defined in your source code. In order to do this, Firebase CLI runs a small tool (let’s call this a harness) that runs and inspects the functions defined in your source code.

Today, there are 2 different ways this process can work:

1. Legacy Firebase CLI includes a small harness that it uses to discover functions defined in your code.
2. Functions SDK includes the harness so it can bootstrap itself and advertise the functions defined in your code.

We made the shift to prefer (2) if possible as a way to have a language-agnostic process for discovering functions defined in the source code.

Unfortunately, we didn’t anticipate how difficult it would be to locate the Firebase Functions SDK package in your source code. We really only considered the simple case where node_modules directory exists inside your functions package. We didn’t foresee complication with using more advanced dependency/build tools like pnpm, nx, and other monorepo utilities.

For now, I’m looking to make a quick patch to address the scenarios raised in here, but I hope we can spend some time in the future to make the overall process more reliable. Stay tuned.

We have exactly the same problem, rolled back from 11.20.0 to 11.17.0 and we are able to deploy once again. For the record we are on Node 16 using Yarn Berry 3.1.1 with workspaces and node-modules nodeLinker.

@taeold Could you please consider reopening this issue?

Hi folks - thanks for escalating this issue. Will make this my priority today.

Thanks for getting on this, @taeold. We all appreciate the snappy response.

I do want to bring up the unfortunate lack of support for monorepos across much of this project. As also detailed in #653, a sizable portion of the community has moved to or is moving towards monorepos, yet firebase-functions seems to not be taking this into account. Considering that Google is one of the shining monorepo examples, this is quite surprising.

This isn’t meant to bash the team, but rather encourage y’all to take into account your user base and prioritize bug fixes and testing on monorepo-related things.

Keep up the good work!

This problem is still happening, downgrading to versions 11.17.0 or 11.22.0 or latest is currently not working for us, can someone please check this? Its a development blocker since deployment is impossible AND emulators won’t start.

Same error in 11.20.0

11.20.0 does not contain the fixes merged above.

I can confirm that running npm install from within the functions dir did allow me to run firebase emulators:start --only functions 🎉

I’m not sure about others here but my project is written in TS and a build step outputs JS to functions dir, but does not copy over or install deps from that package.json. Quick solution for now is to add another task to my build or test scripts to ensure npm install happens inside that dir.

Seems as though the way firebase-tools determines the sdkVersion changed a lot to work with yarn workspaces https://github.com/firebase/firebase-tools/commit/355e9b986ed4c947757666c08b128cf22e5cb092#diff-0342df3e11dd503fdca5b2e49395567874f9a9438f96eb24431773d8465db01f

The previous way (using spawn.sync("npm", ["list", "firebase-functions", "--json=true"]) relied on npm’s resolver to keep looking upward and in my case always found the dep installed in the parent dir.

Tagging @taeold as the dev who committed that change - hopefully this thread will help others and we can get a more robust solution 🙏

I could fix this issue just by downgrading the version of firebase tools to v10.0.0 using: npm install -g firebase-tools@10.0.0

I have just updated to 11.27.0 and am still seeing this error.

@taeold

We have encountered this issue where randomly we are able to deploy our gen v2 function and randomly fail with User code failed to load. Cannot determine backend specification error message.

Deeper investigation shows the issue is not having the .yaml file which in turn resulted in an http call which I assumes generates that yaml file.

that http call has a timeout of 30 sec, if the yaml generation was completed under 30 sec our function would deploy otherwise fail…

attaching code snippet from the firebase-tools@11.24.1 sdk

async function detectFromPort(port, project, runtime, timeout = 30000) {
    let res;
    const timedOut = new Promise((resolve, reject) => {
        setTimeout(() => {
            reject(new error_1.FirebaseError("User code failed to load. Cannot determine backend specification"));
        }, timeout);
    });
    while (true) {
        try {
            res = await Promise.race([(0, node_fetch_1.default)(`http://127.0.0.1:${port}/__/functions.yaml`), timedOut]);
            break;
        }
        catch (err) {
            if ((err === null || err === void 0 ? void 0 : err.code) === "ECONNREFUSED") {
                continue;
            }
            throw err;
        }
    }
    const text = await res.text();
    logger_1.logger.debug("Got response from /__/functions.yaml", text);
    let parsed;
    try {
        parsed = yaml.load(text);
    }
    catch (err) {
        logger_1.logger.debug("Failed to parse functions.yaml", err);
        throw new error_1.FirebaseError(`Failed to load function definition from source: ${text}`);
    }
    return yamlToBuild(parsed, project, api.functionsDefaultRegion, runtime);
}

Would appreciate a quick fix for this issue. Adam.

I can confirm that v11.22.0 fixes this issue as described in #5448

v11.19.0 is the version with the breaking changes. So downgrading to 11.18.0 should be fine. The workaround for now (if possible for your situation) is to cd into the functions dir and run npm/yarn install. The changes @taeold has made which closed this issue ARE NOT RELEASED YET, they didn’t make it into v11.20.0 so the workaround of installing node_modules inside the functions dir will still work for you in v11.19.0 and even v11.20.0.

However, even with the changes applied to my codebase, this workaround did not work and neither does “normal” deployment commands. So I do believe this issue should be reopened @taeold. Thanks

I have the same problem with my Nuxt app. To avoid it, I delete the node_models folder in the output directory and reinstall it.

$ NITRO_PRESET=firebase npm run build && cd .output/server && rm -rf node_modules && npm i && cd ../../ && firebase login && firebase deploy

After that everything works normally.

Node v16.18.1 Firebase v11.16.1 Ubuntu 22.04

@ibnzahoor98 downgrading to such an older version is not really a long term fix imo. Plus this issue has been fixed in v11.22.0 as I outlined above.

I’d recommend going back to this version to determine if this issue is the same cause or is a regression since then. I’ve kept up with the latest version and this error has not resurfaced so you may be experiencing a different problem with the same error message. And also try to delete node modules folder completely and even the package-lock or yarn-lock file too, and ensure the node version is modern.

I can confirm that v11.22.0 fixes this issue as described in #5448

As mentionned in #5482, I get following error when using v11.22.0 :

Error: Failed to find location of Firebase Functions SDK. Please file a bug on Github (https://github.com/firebase/firebase-tools/).

In my case, the deploy works on 11.20.0 version of firebase-tools, but only when I add the --debug flag to the command $ firebase --debug deploy --only functions Our functions are in TS in an Nx monorepo but they have their own package.json, etc.

Same error in 11.20.0

That’s fair. But if you already have an npm cache populated from another install then it won’t take long at all, or rsync or cp the whole node modules or even symlink might work. At least for me as all I needed it for was the tests cos firebase will run the install anyway when you deploy

I’m also seeing the same on 11.19.0 but was working fine on 11.18.0. Also happens when running emulators:start and not just deploy. I’ve found the code which it differentiates… running firebase emulators:start --only functions --debug on 11.18.0 outputs:

[2022-12-21T03:49:26.597Z] Validating nodejs source
[2022-12-21T03:49:26.923Z] checkFunctionsSDKVersion encountered error:Invalid Version:  {}
[2022-12-21T03:49:26.924Z] > [functions] package.json contents: {
  "name": "functions",
  "engines": {
    "node": "16"
  },
  "private": true,
  "type": "module",
  "version": "5.2.0",
  "dependencies": {
    "@google-cloud/functions-framework": "3.1.2",
    "array-flatten": "3.0.0",
    "firebase": "9.15.0",
    "firebase-admin": "11.4.0",
    "firebase-functions": "4.1.1",
    "lodash.merge": "4.6.2",
    "node-fetch": "3.3.0",
    "nodemailer": "6.8.0"
  }
}
[2022-12-21T03:49:26.924Z] Building nodejs source
[2022-12-21T03:49:26.925Z] Analyzing nodejs backend spec
[2022-12-21T03:49:27.523Z] Could not parse firebase-functions version '' into semver. Falling back to parseTriggers.
✔  functions: Loaded functions definitions from source: ...

Running on v11.19.0 outputs:

[2022-12-21T04:03:24.281Z] Validating nodejs source
[2022-12-21T04:03:24.585Z] > [functions] package.json contents: {
  "name": "functions",
  "engines": {
    "node": "16"
  },
  "private": true,
  "type": "module",
  "version": "5.2.0",
  "dependencies": {
    "@google-cloud/functions-framework": "3.1.2",
    "array-flatten": "3.0.0",
    "firebase": "9.15.0",
    "firebase-admin": "11.4.0",
    "firebase-functions": "4.1.1",
    "lodash.merge": "4.6.2",
    "node-fetch": "3.3.0",
    "nodemailer": "6.8.0"
  }
}
[2022-12-21T04:03:24.586Z] Building nodejs source
[2022-12-21T04:03:24.591Z] Could not find functions.yaml. Must use http discovery
[2022-12-21T04:03:24.600Z] Error: spawn ./node_modules/.bin/firebase-functions ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
    at onErrorNT (node:internal/child_process:485:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)

Error: An unexpected error has occurred

So you can see the working version says “Could not parse firebase-functions version ‘’ into semver. Falling back to parseTriggers” which is logged from here

Which is only happening when this.sdkVersion is not valid… v11.18.0: this.sdkVersion === “” v11.19.0: this.sdkVersion === 4.1.1 Change happened here - https://github.com/firebase/firebase-tools/commit/355e9b986ed4c947757666c08b128cf22e5cb092#diff-88e84b107a8f48a6a783275f2cef81ea7f269efbc4d2c22906217d0be811d973

This is the cause of then hitting let discovered = await discovery.detectFromYaml(this.sourceDir, this.projectId, this.runtime); and that function tries to rad the functions.yaml file which is not found causing that message:

// https://github.com/firebase/firebase-tools/blob/355e9b986ed4c947757666c08b128cf22e5cb092/src/deploy/functions/runtimes/discovery/index.ts#L43
try {
        text = await exports.readFileAsync(path.join(directory, "functions.yaml"), "utf8");
    }
    catch (err) {
        if (err.code === "ENOENT") {
            logger_1.logger.debug("Could not find functions.yaml. Must use http discovery");
        }
        else {
            logger_1.logger.debug("Unexpected error looking for functions.yaml file:", err);
        }
        return;
    }

Just rolling back to 10.8.0 has solved it for me, lost 2 days of debugging a pipeline haha

@colerogers I use npm. I found a solution - upgrading from Node v14 to v16 helped to solve the problem.