nodejs-docker: Error installing node.js: The Node.js binary could not be verified.

I confirm: Node 14.17.0 is having that same issue. Note: Node 14.16.0 works fine.

The issue is possibly that the KEYS file in this repository has not been updated to contain @danielleadams’s new key (updated in January, https://github.com/nodejs/node/pull/36793).

The Node.js project has recently started to maintain https://github.com/nodejs/release-keys/tree/main/keys as a source of truth for our release keys, which might be a useful resource for keeping this repository in sync with Node.js key updates.

FYI, fix is almost ready. We plan to release it to prod tomorrow 4/5/2022.

according to the nodejs releases, 12.22.7 has been released at 2021/10/12, so I guess that’s where it got broken.

Any releases signed by @danielleadams’s key will not verify until the keys are updated in this repository, which include the latest security releases released on 2021-10-12 (v12.22.7, v14.18.1, and v16.11.1).

How is this still an issue, trying to use node v16 and getting this, trying to use node LTS v18 and getting glib errors https://github.com/GoogleCloudPlatform/buildpacks/issues/198

almost impossible to use app engine for anything critical these days

@jinglundong

Thanks for sharing.

We’ve used that pinned solution in the past and have, indeed, already deployed such a fix today.

I recognize that this isn’t your problem per se, but I do want to explicitly point out the fact that a recommendation to fix a deployment issue in one Google Infrastructure component actually introduces a new issue in a separate Google infrastructure component.

Google Cloud Functions deployments will raise an error if a package.json when engine is not a whole number (anything other than 12,14,16, etc…) … but the solution recommended here requires specifying an engine value other than the above (e.g. 16.17.1).

The result is that, without additional work, app engine deploys succeed and function deployments fail. This is in a monorepo where splitting the systems is impractical, even if possible.

To support both, our build systems basically do a sed to modify the pinned version for AppEngine to just a whole number for Functions.

In any case, thansk for hearing me out.

We really appreciate the work y’all do to keep GCP stable and performant.

It should probably remain open until ongoing fix is found. On Fri, 3 Dec 2021 at 12:50, Oleksii Kachan @.***> wrote:

works for me now with 14.18.2. I think this issue is good to close

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/GoogleCloudPlatform/nodejs-docker/issues/214#issuecomment-985147480, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADDR7UZ5ZJQ5BNUBPEHMPDUPAO4VANCNFSM4MI6GDGA .

Same as what @brunogabuzomeu said, we tempoarily solved it by using =16.10.0 which still works.

tl;dr: I can take a look at the new key and probably update it. To mitigate the problem, please pin it to the previous patch version of Node.js 16.17.x or Node.js 16.17.1. https://nodejs.org/en/download/releases/

I haven’t looked into the exact problem, but I’d suspect that it’s broken because there’s a new pgp key in Node.js 16.18. This means our builder (or nodejs itself) is not entirely backward compatible between minor version upgrades.

This is a known issue where we don’t really have a short term automation on our side. The best option in to pin to a patch version of nodejs version. Because new pgp keys are getting added along with new patch version of nodejs. We don’t have a way to detect that, or block anyone from picking it up immediately when they have something like 16.x.

We are still working on a project to improve our build system. Once that’s complete, we should be able to control this behavior - we will qualify the new minor or patch version before the builder using it.

This started failing again today for node 16.x. Observed that node published a release circa 11AM today.

Step #2: gpg: Signature made Wed Oct 12 14:48:44 2022 UTC using RSA key ID 94EC4689 gpg: Can't check signature: public key not found
Step #2: The Node.js binary could not be verified.
Step #2: This means it may not be an officially released Node.js binary
Step #2: or may have been tampered with.
Step #2: 
Step #2: Aborting the installation.
Step #2: 
Step #2: The installation can be forced using the --ignore-verification-failure
Step #2: flag. However, it is strongly recommended that you install a version
Step #2: of Node.js that can be verified.
Step #2: 
Step #2: Node installation failed: /opt/gcp/runtime/bootstrap_node exited with a non-zero exit code: 1
Step #2: error building image: error building stage: waiting for process to exit: exit status 1
Finished Step #2

What can we do to be insulated from this error without having to pin versions in our application itself?

Occurring in GAE flex again with 14.x set.

Step #1: INFO[0047] RUN /usr/local/bin/install_node '14.x'       
Step #1: INFO[0047] cmd: /bin/sh                                 
Step #1: INFO[0047] args: [-c /usr/local/bin/install_node '14.x'] 
Step #1:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #1:                                  Dload  Upload   Total   Spent    Left  Speed
100 32.3M  100 32.3M    0     0  43.1M      0 --:--:-- --:--:-- --:--:-- 43.2M
Step #1:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #1:                                  Dload  Upload   Total   Spent    Left  Speed
100  3835  100  3835    0     0  24299      0 --:--:-- --:--:-- --:--:-- 24426
Step #1: gpg: Signature made Tue May 11 23:13:29 2021 UTC using RSA key ID 643B6201 gpg: Can't check signature: public key not found
Step #1: The Node.js binary could not be verified.
Step #1: This means it may not be an officially released Node.js binary
Step #1: or may have been tampered with.
Step #1: 
Step #1: Aborting the installation.
Step #1: 
Step #1: The installation can be forced using the --ignore-verification-failure
Step #1: flag. However, it is strongly recommended that you install a version
Step #1: of Node.js that can be verified.
Step #1: 
Step #1: Node installation failed: /opt/gcp/runtime/bootstrap_node exited with a non-zero exit code: 1
Step #1: error building image: error building stage: waiting for process to exit: exit status 1
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/kaniko-project/executor@sha256:f87c11770a4d3ed33436508d206c584812cd656e6ed08eda1cff5c1ee44f5870" failed: step exited with non-zero status: 1

Edit: setting the package engine version to the last successfully deployed version (14.6.1) instead of 14.x works around the issue for now.

Hello @plegner

in our case, it was the same as for @aiimtechdev.

We had broken deployment to App Engine because it installed Node 14.10.0 as we had "node": "14.x" in the package.json. Seems like the latest release has some issues.

I fixed deploy by changing the engine to a fixed version:

"engines": {
    "node": "14.9"
}

If you are using 12.x - try some of the previous version that worked.

Obviously it does not solve the overall issue with the latest release but at least get things working.

Hope it will help!

Hi folks - We hit this issue as well with our NodeJS app engine deployment. We were running 16.x. Pinning did end up working at a pin of 16.17.x. For future devs that hit this issue, make sure you pin to the precise version - 16.17.x NOT ^16.17.x

Here’s a trace:

Step #2: INFO[0017] Taking snapshot of full filesystem...
Step #2: INFO[0024] Using files from context: [/workspace]       
Step #2: INFO[0024] COPY . /app/                                 
Step #2: INFO[0024] Taking snapshot of files...
Step #2: INFO[0026] RUN /usr/local/bin/install_node '^16.17.x'   
Step #2: INFO[0026] cmd: /bin/sh                                 
Step #2: INFO[0026] args: [-c /usr/local/bin/install_node '^16.17.x'] 
Step #2:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #2:                                  Dload  Upload   Total   Spent    Left  Speed
100 32.3M  100 32.3M    0     0  90.4M      0 --:--:-- --:--:-- --:--:-- 90.6M
Step #2:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #2:                                  Dload  Upload   Total   Spent    Left  Speed
100  4035  100  4035    0     0  46219      0 --:--:-- --:--:-- --:--:-- 46379
Step #2: gpg: Signature made Wed Oct 12 14:48:44 2022 UTC using RSA key ID 94EC4689 gpg: Can't check signature: public key not found
Step #2: The Node.js binary could not be verified.
Step #2: This means it may not be an officially released Node.js binary
Step #2: or may have been tampered with.
Step #2: 
Step #2: Aborting the installation.
Step #2: 
Step #2: The installation can be forced using the --ignore-verification-failure
Step #2: flag. However, it is strongly recommended that you install a version
Step #2: of Node.js that can be verified.
Step #2: 
Step #2: Node installation failed: /opt/gcp/runtime/bootstrap_node exited with a non-zero exit code: 1
Step #2: error building image: error building stage: waiting for process to exit: exit status 1
Finished Step #2
ERROR
ERROR: build step 2 "gcr.io/kaniko-project/executor@sha256:f87c11770a4d3ed33436508d206c584812cd656e6ed08eda1cff5c1ee44f5870" failed: step exited with non-zero status: 1

Experiencing this error today with Node 18.5.0.

Setting 14.18.0 works as a short term solution – unfortunately, if you are deploying, say, cloud functions from the same project, "engines": {"node": "14.18.0"} is invalid according to a separate Google technology platform.

Error: package.json in functions directory has an engines field which is unsupported. Valid choices are: {"node": 10|12|14|16}

In addition to my signing key, whoever maintains this repository may also want to add the keys for @ruyadorno (https://github.com/nodejs/node/pull/34628) and @danielleadams (https://github.com/nodejs/node/pull/35545) who have also been recently onboarded as releasers and may sign future releases: https://github.com/nodejs/node/blob/master/README.md#release-keys

@jinglundong is there any reason why it would not even try and run an install of the node version specifed in the package.json? We have our engine specified and have tried pinning to 16.17.1 but it doesn’t even attempt an install. Please see the output below:

FETCHSOURCE
BUILD
Starting Step #0 - "fetcher"
Step #0 - "fetcher": Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Step #0 - "fetcher": Fetching manifest gs://staging.****************.appspot.com/ae/5d15c126-2759-4b47-926a-5ee0ae1a4f93/manifest.json.
Step #0 - "fetcher": Processing 112 files.
Step #0 - "fetcher": ******************************************************
Step #0 - "fetcher": Status:                      SUCCESS
Step #0 - "fetcher": Started:                     2022-11-08T04:47:33Z
Step #0 - "fetcher": Completed:                   2022-11-08T04:47:33Z
Step #0 - "fetcher": Requested workers:    200
Step #0 - "fetcher": Actual workers:       112
Step #0 - "fetcher": Total files:          112
Step #0 - "fetcher": Total retries:          0
Step #0 - "fetcher": GCS timeouts:           0
Step #0 - "fetcher": MiB downloaded:         0.49 MiB
Step #0 - "fetcher": MiB/s throughput:       2.57 MiB/s
Step #0 - "fetcher": Time for manifest:    104.01 ms
Step #0 - "fetcher": Total time:             0.30 s
Step #0 - "fetcher": ******************************************************
Finished Step #0 - "fetcher"
Starting Step #1
Step #1: Pulling image: gcr.io/gcp-runtimes/nodejs/gen-dockerfile@sha256:8e2b8eef87fae1bc4b80206cc84139a52973433daafef5ba30ae84f2f85d01b6
Step #1: gcr.io/gcp-runtimes/nodejs/gen-dockerfile@sha256:8e2b8eef87fae1bc4b80206cc84139a52973433daafef5ba30ae84f2f85d01b6: Pulling from gcp-runtimes/nodejs/gen-dockerfile
Step #1: Digest: sha256:8e2b8eef87fae1bc4b80206cc84139a52973433daafef5ba30ae84f2f85d01b6
Step #1: Status: Downloaded newer image for gcr.io/gcp-runtimes/nodejs/gen-dockerfile@sha256:8e2b8eef87fae1bc4b80206cc84139a52973433daafef5ba30ae84f2f85d01b6
Step #1: gcr.io/gcp-runtimes/nodejs/gen-dockerfile@sha256:8e2b8eef87fae1bc4b80206cc84139a52973433daafef5ba30ae84f2f85d01b6
Step #1: Checking for Node.js.
Finished Step #1
Starting Step #2
Step #2: Already have image (with digest): gcr.io/kaniko-project/executor@sha256:f87c11770a4d3ed33436508d206c584812cd656e6ed08eda1cff5c1ee44f5870
Step #2: INFO[0000] Removing ignored files from build context: [node_modules .dockerignore Dockerfile npm-*****.log yarn-error.log .git .hg .svn app.yaml] 
Step #2: INFO[0000] Downloading base image gcr.io/google-appengine/nodejs@sha256:f4979bc91ed28ae6fc489b5377fb87d628263fe7a7da971b134c03aff47327b2
Step #2: INFO[0017] Taking snapshot of full filesystem...
Step #2: INFO[0024] Using files from context: [/workspace]       
Step #2: INFO[0024] COPY . /app/                                 
Step #2: INFO[0024] RUN NODE_ENV=*********** yarn install ||   ((if [ -f yarn-error.log ]; then       cat yarn-error.log;     fi) && false) 
Step #2: INFO[0024] cmd: /bin/sh                                 
Step #2: INFO[0024] args: [-c NODE_ENV=*********** yarn install ||   ((if [ -f yarn-error.log ]; then       cat yarn-error.log;     fi) && false)]
Step #2: yarn install v1.22.19
Step #2: [1/5] Validating package.json...
Step #2: error *************-api@1.0.0: The engine "node" is incompatible with this module. Expected version "16.17.1". Got "12.19.0"
Step #2: error Found incompatible module

We updated the key at https://github.com/GoogleCloudPlatform/nodejs-docker/pull/254 and https://github.com/GoogleCloudPlatform/nodejs-docker/pull/256. I verified this in prod with a 16.x app that got resolved to 16.18.1. Please let us know if you have any other questions.

To be clear, this doesn’t solve the problem of 18.x. Node.js 18 support requires a bigger effort. We have it planned in the first half of 2023.

I experienced this issue today. I couldn’t see which version the system was attempting to install anywhere in the logs.

In the failing case, I saw:

Step #2: INFO[0105] RUN /usr/local/bin/install_node '14.x'
Step #2: INFO[0105] cmd: /bin/sh
Step #2: INFO[0105] args: [-c /usr/local/bin/install_node '14.x']
Step #2:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #2:                                  Dload  Upload   Total   Spent    Left  Speed
100 32.9M  100 32.9M    0     0  77.6M      0 --:--:-- --:--:-- --:--:-- 77.7M
Step #2:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #2:                                  Dload  Upload   Total   Spent    Left  Speed
100  3835  100  3835    0     0  35435      0 --:--:-- --:--:-- --:--:-- 35841
Step #2: gpg: Signature made Fri Sep 23 16:05:56 2022 UTC using RSA key ID E57C7D57 gpg: Can't check signature: public key not found
Step #2: The Node.js binary could not be verified.
Step #2: This means it may not be an officially released Node.js binary
Step #2: or may have been tampered with.

Looking at the signature date of Sep 23rd and our restriction to Node 14.x, we can infer that it’s 14.20.1 as it was released on that day, but it would have been helpful if the logs showed which version was being attempted. Better yet, it would be helpful if the system didn’t try to use unsupported versions at all.

For reference, in a successful installation (package.json engine version restricted to 14.19), the immediate next step shows the full installed version number:

Step #2: INFO[0102] RUN /usr/local/bin/install_node '14.19'
Step #2: INFO[0102] cmd: /bin/sh
Step #2: INFO[0102] args: [-c /usr/local/bin/install_node '14.19']
Step #2:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #2:                                  Dload  Upload   Total   Spent    Left  Speed
100 32.9M  100 32.9M    0     0  59.2M      0 --:--:-- --:--:-- --:--:-- 59.1M
Step #2:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #2:                                  Dload  Upload   Total   Spent    Left  Speed
100  3835  100  3835    0     0  19242      0 --:--:-- --:--:-- --:--:-- 19271
Step #2: node-v14.19.3-linux-x64.tar.gz: OK
Step #2: Installed Node.js v14.19.3

Is there an explicit list of which specific minor versions of Node aren’t going to trigger this problem?

That is correct. I’m listing a few ideas below following my preferred order, but I haven’t had a chance to figure out the details. Please advice if anyone has better solutions.

1. We have a plan to simplify how we build Flex applications. I believe it’s going to solve this problem, or at least be able to patch it much faster. This is the actual fix, but it will likely take around two quarters to develop.
2. If new keys are added and announced before they are being used (which would cause the GPG check failure), we are happy to monitor that list of keys more closely. This is a short term solution.
3. We may have a way to detach from latest, and pin to a relatively new node version. Not always using the latest version of node may cause problems for some users too.

Just to note, this is going to have to be re-opened the next time a new GPG key is added, the problem wasn’t solved.

@jinglundong I have almost the same configuration:

  "engines": {
    "node": "14.x",
    "npm": ">=6.0.0"
  }

runtime: nodejs

env: flex

# Resources configuration (for each server)
resources:
  cpu: 1
  memory_gb: 2
  disk_size_gb: 10

# Configuration of automatic scaling
# block max_num_instance (3)
automatic_scaling:
  min_num_instances: 1
  max_num_instances: 1
  cool_down_period_sec: 180
  cpu_utilization:
    target_utilization: 0.6
# [END app_yaml]

Step #1: yarn install v1.22.18
Step #1: [1/5] Validating package.json...
Step #1: error *************@1.5.30: The engine "node" is incompatible with this module. Expected version "14.x". Got "12.19.0"
Step #1: error Found incompatible module.
Step #1: info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Step #1: error building image: error building stage: waiting for process to exit: exit status 1
Finished Step #1

gcloud app deploy app-dev.yaml --project=*******

@niciliketo the 😕 was for the maintainers, sorry. Disappointing this hasn’t been taken care of yet?

I am sseing a similar issue for 16.4.x It seems to have started from 2022-03-26

Step #1: INFO[0022] COPY . /app/                                 
Step #1: INFO[0022] Taking snapshot of files...                  
Step #1: INFO[0023] RUN /usr/local/bin/install_node '>=16.4.x'   
Step #1: INFO[0023] cmd: /bin/sh                                 
Step #1: INFO[0023] args: [-c /usr/local/bin/install_node '>=16.4.x'] 
Step #1:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #1:                                  Dload  Upload   Total   Spent    Left  Speed
100 40.9M  100 40.9M    0     0   102M      0 --:--:-- --:--:-- --:--:--  102M
Step #1:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Step #1:                                  Dload  Upload   Total   Spent    Left  Speed
100  4009  100  4009    0     0  41326      0 --:--:-- --:--:-- --:--:-- 41760
Step #1: gpg: Signature made Tue Mar 22 14:38:58 2022 UTC using RSA key ID E57C7D57 gpg: Can't check signature: public key not found
Step #1: The Node.js binary could not be verified.
Step #1: This means it may not be an officially released Node.js binary
Step #1: or may have been tampered with.
Step #1: 
Step #1: Aborting the installation.
Step #1: 
Step #1: The installation can be forced using the --ignore-verification-failure
Step #1: flag. However, it is strongly recommended that you install a version
Step #1: of Node.js that can be verified.

@geiszla your issue seems irrelevant to this issue. The fix mentioned in above is about using latest version fails due to GPG key mismatch.

Your issue is about mismatch of runtime node version and engine version specified in package.json.

works for me now with 14.18.2. I think this issue is good to close

@chiefy ah, thanks a lot! I haven’t check repository close enough 🤦🏽‍♂️ I’ll keep in my mind in case we have next update.

@pete-beresford-bzy, Hmm…I extended my key back in March but it seems it hasn’t properly propagated/synced with the servers - I’ll try resyncing/resubmitting.

I’ve the same problem using Google App Engine Flex & Standard: if I downgrade to 14.16.x using "engines": { "node": "14.16.x" } in my package.json I’d work but if I deploy the package in another environment using the standard environment it would again result in an error-message because of the wrong version:

The 'package.json' configuration file must specify a supported nodejs14 version that is compatible with the runtime specified in the deployment. Please pin your application to a compatible major version of the runtime via the 'engines.node' property. Supported 'engines.node' values for the runtime 'nodejs14' are: [14.x.x, 14.X.X, 14.x, 14.X, 14.*.*, 14.*, ~14, ~14.0, ~14.0.0, 14, ^14, ^14.0, ^14.0.0, >=14, >=14.0, >=14.0.0, >14.0, >14.0.0, >=1, >=1.0, >=1.0.0, >1, >1.0, >1.0.0, >=2, >=2.0, >=2.0.0, >2, >2.0, >2.0.0, >=3, >=3.0, >=3.0.0, >3, >3.0, >3.0.0, >=4, >=4.0, >=4.0.0, >4, >4.0, >4.0.0, >=5, >=5.0, >=5.0.0, >5, >5.0, >5.0.0, >=6, >=6.0, >=6.0.0, >6, >6.0, >6.0.0, >=7, >=7.0, >=7.0.0, >7, >7.0, >7.0.0, >=8, >=8.0, >=8.0.0, >8, >8.0, >8.0.0, >=9, >=9.0, >=9.0.0, >9, >9.0, >9.0.0, >=10, >=10.0, >=10.0.0, >10, >10.0, >10.0.0, >=11, >=11.0, >=11.0.0, >11, >11.0, >11.0.0, >=12, >=12.0, >=12.0.0, >12, >12.0, >12.0.0, >=13, >=13.0, >=13.0.0, >13, >13.0, >13.0.0].

So this should be updated in both environments.

This error also occurs with the latest release of Node 10 10.23.0 which was released by @richardlau. His signing key is missing in the KEYS file https://github.com/GoogleCloudPlatform/nodejs-docker/blob/master/runtime-image/contents/KEYS, it was added back in july in the Node repo https://github.com/nodejs/node/pull/34397/files