docker-node: `yarn` not found in `node:alpine`

I think it’s a little bit awkward that changes will be commited to existing tags other than “rc”, “nightly” or similar. In my opinion a version like 9.8.0 should never ever be changed. When changes have to be made it’s 9.8.1. Many many ci pipelines will be broken today with many many hours of work of devops to find the solution.

This repo needs better unit tests 🍵

I would be very interested in reading a blog post on how this mistake happened and what steps you’ve taken to ensure that it won’t happen again.

The fix seems to have made its way through the system. I’ve just had a GitLab CI build pass using the node:alpine image.

A version label should be cast in stone, particularly for something as widely used as this. Lots of wasted hours and heads scratched over this today.

I wrote up a blog post to explain how/why this happened, why it’s not completely unexpected and can happen again, and what you can do to protect against it in future: https://renovateapp.com/blog/docker-mutable-tags

In summary:

1. Docker tags can change at any time. They are mutable and that is by design, not fault
2. Even a Docker tag that looks like semver (e.g. node:8.10.0) still has the properties of a tag and can be changed/updated at any time, like happened here
3. There are valid reasons for the image a tag points to to be changed (such as refactoring the Dockerfile or a security patch to the base), so it’s not a good idea to insist that a tag that looks like a semver should never be changed
4. The only way to ensure your build is reproducible/consistent is to use Docker digests as the identifier instead of the tag
5. You can conveniently still include a Docker tag in the digest for human readability, e.g. node:8.10.0-alpine@sha256:a55d3e87802b2a8464b3bfc1f8c3c409f89e9b70a31f1dccce70bd146501f1a0
6. If you were using digests for your node base images instead of tags, you not only wouldn’t have “accidentally” upgraded to the new broken image in the first place, and even if you had updated on purpose then you could have immediately rolled back the digest in your Dockerfile once you discovered the break, instead of waiting hours
7. Beyond that, these digests are a little “developer-unfriendly” to update by hand, but open source tools to automate it exist

This was caused by this commit: https://github.com/nodejs/docker-node/commit/ae26d217bca933e72ac5f8de946681c1b5f8b37e

Given the way the symlink is done mkdir -p /opt/yarn should have been deleted

I’m happy we were not the only one experiencing this issue 😅 , reverting to a earlier tag was our solution but kudos to the community on this form for providing suggestions.

I think the whole world ended up here when they realized their builds were not passing lol!

This affects 9.8.0, 9.8, 9, latest, 8.10.0, 8.10, 8, carbon, 6.13.1, 6.13, 6, boron, 4.8.7, 4.8, 4, argon and all variants (alpine, slim, onbuild, wheezy, stretch)

You need to wait for the image to be published. It’s currently being built

A docker image is an artifact of its own, and thus needs it own version. The image version may not need to be a full semver though, only a build increment. node:alpine-8.6.0_16 for the 17th release of node-alpine-8.6.0 might be sufficient. node:alpine-8-6 and node:alpine-8 might still be mutable “tags”.

@davidblurton As I see it there were few issues:

1. We didn’t have a basic test of yarn which would have caught this early (fixed in #650)
2. We should not have immediately pushed the changes to the docker hub. Instead, the changes to how yarn is installed/linked should have been grouped with a Node.js (and therefor docker tag) update.

For me the issue was in using singlequotes in Dockerfile CMD command

CMD ['yarn', 'start'] // Wrong
CMD ["yarn", "start"] // Correct

There’s a good reason for tags not being immutable. One of the main benefit of Docker is that you inherit security fixes to underlying images. Sure there’s a chance that a change to the base image might break a build but it’s a trade-off that I personally gladly take to get the security benefits.

@n1ru4l depends on how you specify the semver for the node images: should it be inline with the nodejs version? IMHO: yes

but then you might also want to have security patches for the underlying OS for this nodejs version to somehow show up (automatically)?

@PeterDaveHello @chorrell it seems that this PR: https://github.com/nodejs/docker-node/pull/647 is not backward-compatible and broke Docker Images (e.g. 8.10.0-alpine)

All of our build pipelines are now stuck due to errors such as:

/bin/sh: yarn: not found

npm ERR! Refusing to delete /usr/local/bin/yarn: /opt/yarn/bin/yarn symlink target is not controlled by npm /usr/local/bin

Currently, as a workaround one can install yarn via apk.

Example:

# in Dockerfile
FROM node:8.10.0-alpine

RUN apk add --no-cache git openssl ca-certificates yarn

IMHO tagging model of official node Docker images is broken. It must not be possible to change image of already created image, instead new tag should be used.

Latest working version seems to be node:8.9.4 and node:8.9.4-alpine

@chorrell @SimenB please have a look ^_^ I put so much effort into it, I would be glad to contribute to the project.

Adding the following to my Dockerfile as a temporary fix helped. Remember YMMV! RUN mkdir -p /opt/yarn/bin && ln -s /opt/yarn/yarn-v1.5.1/bin/yarn /opt/yarn/bin/yarn

latest is affected I think:

The link /usr/local/bin/yarn is broken, it targets /opt/yarn/bin/yarn which doesn’t exist. But /opt/yarn/yarn-v1.5.1 exists.

root@5706d42db0c6:/# ls /usr/local/
CHANGELOG.md  LICENSE  README.md  bin  etc  games  include  lib  man  sbin  share  src
root@5706d42db0c6:/# ls /usr/local/bin
node  nodejs  npm  npx  yarn  yarnpkg
root@5706d42db0c6:/# /usr/local/bin/yarn
bash: /usr/local/bin/yarn: No such file or directory
root@5706d42db0c6:/# ls /usr/local/bin
node  nodejs  npm  npx  yarn  yarnpkg
root@5706d42db0c6:/# stat /usr/local/bin/yarn
  File: '/usr/local/bin/yarn' -> '/opt/yarn/bin/yarn'
  Size: 18              Blocks: 0          IO Block: 4096   symbolic link
Device: 31h/49d Inode: 62          Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-03-16 10:19:06.000000000 +0000
Modify: 2018-03-16 10:19:06.000000000 +0000
Change: 2018-03-16 11:13:53.243600600 +0000
 Birth: -
root@5706d42db0c6:/# /usr/local/bin/yarn
bash: /usr/local/bin/yarn: No such file or directory
root@5706d42db0c6:/# /usr/local/bin/n
node    nodejs  npm     npx
root@5706d42db0c6:/# /usr/local/bin/yarn
bash: /usr/local/bin/yarn: No such file or directory
root@5706d42db0c6:/# /opt/yarn/bin/yarn
bash: /opt/yarn/bin/yarn: No such file or directory
root@5706d42db0c6:/# ls /opt/yarn
yarn-v1.5.1
root@5706d42db0c6:/# ls /opt/yarn/yarn-v1.5.1
LICENSE  README.md  bin  lib  package.json
root@5706d42db0c6:/# ls /opt/yarn/yarn-v1.5.1

@SimenB @LaurentGoderre - PR with changes to prevent this sort of thing happening again - https://github.com/nodejs/docker-node/pull/658

This issue broke our pipeline and we did a workaround as follows:

/opt/yarn/yarn-v1.5.1/bin/yarn install

We’re fixing it in #650

The image node:9.7-alpine can be used as a workaround until this is fixed.

I’m happy we were not the only one experiencing this issue 😅 , reverting to a earlier tag was our solution but kudos to the community on this form for providing suggestions.

Also note that the tag is updated automatically by docker hub to account for security fixes in the underlying OS, so the only way to truly lock it down is to use the id of the image, not the tag

IMHO tagging model of official node Docker images is broken. It must not be possible to change image of already created image, instead new tag should be used.

@orfin A bit off topic, but I could not agree more. I believed the one of the selling points of Docker was to create repeatable builds. Currently this is only possible using @<tag> notation in Dockerfile, which is not very user developer friendly.

Please don’t spam down the issue with what images are broken, full list here: https://github.com/nodejs/docker-node/issues/649#issuecomment-373686322 (feel free to correct me if any are missed there, though!)

@cecton thanks!

I’m seeing the same in circleci

Step 6/9 : RUN yarn
 ---> Running in a8731d6619d5
/bin/sh: yarn: not found
The command '/bin/sh -c yarn' returned a non-zero code: 127