buildpacks: gcloud run deploy fails because npm run build is executed automatically

I just wanted to call out that this is a breaking change that could catch a lot of people unawares. It is blocking our deployment flow and it took me a while to find the root cause (when I compared a successful build to a failed one, I don’t even see any difference in buildpack version numbers). Please be more intentional about rolling out breaking changes like this.

This broke us too. We’re using Node.js v16 and Google Cloud Functions for Firebase and our firebase.json looks like this:

{
  "functions": {
    "predeploy": [
      "npm --prefix \"$RESOURCE_DIR\" run build"
    ],
    "source": "functions"
  }
}

So when we run firebase deploy it does the npm run build step. Doing it in the buildpack is a waste of time and, more importantly, broke us because certain types were not available in Cloud Build.

We’ve been deploying this way for years so this is an undocumented breaking change. For now I am going to rename my build script to compile but IMO this should be rolled back.

Hi Firebase dev here! If anyone is using Cloud Functions for Firebase and firebase-deploy (like @samatcolumn), we patched our CLI to disable this feature a few weeks back in firebase-tools v11.27.0. If you upgrade to at least that version, this error should go away without any additional work. Thanks!

@FrozenKiwi We couldn’t get it working with the project.toml either. Adding an empty gcp-build script to the package.json did it though!

It’s unfortunate that this breaking change is not adequately documented

Current workaround is to change the build command in package.json to build:local

Adding GOOGLE_EXPERIMENTAL_NODEJS_NPM_BUILD_ENABLED as a keyword here, for my fellow humans wondering what kind of bad experiment Google is running and spending half a day on it before ending up in this issue 🤦

The docs already call out a gcp-build npm script. I don’t see why calling build needs to also happen?

The workaround here is to drop a file named project.toml into the root of your project with the following content:

[[build.env]]
name = "GOOGLE_NODE_RUN_SCRIPTS"
value = ""

Explanation: GOOGLE_NODE_RUN_SCRIPTS takes a comma separated list of npm scripts to run as part of the build. Providing an empty string, will cause no scripts to run.

Spent way too much time tracking this down. This should’ve 💯 been marked as a breaking change.

@mctrafik et al., thanks for sharing your respective situations.

It takes days. I just had another developer waste 2 days on this.

I want to apologize for the frustrations that this change has caused and I appreciate ya’ll spending the time to give us this feedback. I wanted to share some of the changes we’re making to help with the discovery and debug-ability of this issue:

• We changed the published release notes to be marked as breaking change (GAE, GCF)
• Added error messaging on the Node.js buildpack to help with debugging
• Additional documentation to clarify how to work around the breakage on the Node.js buildpacks documentation

We’ve also identified process and communication gaps:

• Release notes for this change were not published in Cloud Run. In the future, with change this big we will be publishing GCP Buildpack changelogs in Cloud Run Release Notes as well
• The GCP Buildpacks documentation site didn’t have release notes in April when we rolled out this change. We’ve addressed this by introducing a Release Notes section as well for GCP Buildpacks

And if it did, it shouldn’t be in the way that it did.

I agree, ultimately this was a breaking change to the Node.js buildpacks. We underestimated the potential negative impact of this change and we strive to do better in the future. A few things we’re committed to doing:

1. Get more involvement and feedback from the community on major upcoming changes before they happen. For changes that may have a broad impact on the project, I’ll be posting them in the project’s Discussions board to get community feedback
2. Known breaking changes will always have a 1 year notice period before coming into effect. Those will always be communicated via release notes, here on the GitHub project, and also through official Google communication channels (if you’re a customer)

The worst part is that this happens when a new version of node is used.

I might be missing some additional details here, but that shouldn’t be the case. Node runtime versions are decoupled from the actual buildpack behavior

@jama22 I spoke to Google support that said your answer should be considered ‘official’ which is bonkers to me.

Our usecase is that we use an internal registry with private access and all of our build commands require credentials that we will not be sending to Google for security reasons. This means every single team, developer and project requires to try to deploy code to GCP via trial and error until they find the internal logs for buildpack setup and figure out that there’s a check for gcp-build like I had to do. It takes days. I just had another developer waste 2 days on this.

The worst part is that this happens when a new version of node is used. So everyone in my company, when they will slowly be upgrading node versions will over the next few years discover this bug and struggle. Everyone will. This change should not have happened. And if it did, it shouldn’t be in the way that it did.

I.e. if you want to make this change permanent there need to be dozens of guides and an e-mail to all current GCP nodeJs runtime developers notifying us of the breaking change that is SO DIFFICULT to troubleshoot on prod.

It may not matter, but there is some small difference between the empty gcp-build and declaring the GOOGLE_NODE_RUN_SCRIPTS build env var noted over in https://github.com/GoogleCloudPlatform/buildpacks/issues/290. Declaring the env var runs everything with NODE_ENV production; declaring gcp-build still installs dev dependencies, runs the (empty) script, then uninstalls the dev dependencies using npm --prune. Also noted in that issue was a bug that it would then actually launch the app with NODE_ENV development true, which broke things for us, but that has apparently been resolved.

The workaround here is to drop a file named project.toml into the root of your project with the following content:

[[build.env]]
name = "GOOGLE_NODE_RUN_SCRIPTS"
value = ""

Explanation: GOOGLE_NODE_RUN_SCRIPTS takes a comma separated list of npm scripts to run as part of the build. Providing an empty string, will cause no scripts to run.

It works, thanks

Super annoyingly, for anyone else using github actions to deploy… that devs on that team don’t allow any empty values to be passed. So the only workaround in that case is using the empty gcp-build script.

I mentioned this thread in an open PR that’s trying to resolve this issue for anyone interested.

Oof. 🤯 Just spent 2 hours figuring out why our CI/CD pipeline was failing, and then I finally found this issue. I agree with others about marking this as a breaking change.

Renaming my build command in package.json also fixed the problem for me.

@FrozenKiwi We couldn’t get it working with the project.toml either. Adding an empty gcp-build script to the package.json did it though!

Relevant docs: https://cloud.google.com/docs/buildpacks/nodejs#using_google_node_run_scripts

@steveoh the Procfile worked until yesterday for me, now just the Procfile is not enough.

With both, the project.toml and Procfile, it finally deploys again.

Maybe that depends on which “default” npm-scripts are used in package.json?

• start disabled with Procfile
• build disabled with project.toml

both scripts are not meant for the server in my case

The workaround here is to drop a file named project.toml into the root of your project with the following content:

[[build.env]]
name = "GOOGLE_NODE_RUN_SCRIPTS"
value = ""

Explanation: GOOGLE_NODE_RUN_SCRIPTS takes a comma separated list of npm scripts to run as part of the build. Providing an empty string, will cause no scripts to run.

Thanks, i can confirm that our deployment is working again after adding the file. From what i understood this is a workaround which will be removed/solved differently in the future, right?

Now i have to know internals from the buildpack in my project even though i just run “gcloud run deploy” for deployment and not building the software.(even though the container is internally build i know)