ruby: Alpine 3.14 images can fail on Docker versions older than 20.10

We run a slightly old Container-Optimized OS from Google (13310.1041.24) on thousands of Google N2D machines, and after the latest updates using Alpine v3.14, we noticed all sorts of strange failures:

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.13 bundle --version
Bundler version 2.2.22
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.14 bundle --version
`/root` is not writable.
Bundler will use `/tmp/bundler20210709-1-4lza5z1' as your home directory temporarily.
Bundler version 2.2.22
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:2.7
irb(main):001:0>
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:2.7-alpine
Traceback (most recent call last):
	6: from /usr/local/bin/irb:23:in `<main>'
	5: from /usr/local/bin/irb:23:in `load'
	4: from /usr/local/lib/ruby/gems/2.7.0/gems/irb-1.2.6/exe/irb:11:in `<top (required)>'
	3: from /usr/local/lib/ruby/2.7.0/irb.rb:393:in `start'
	2: from /usr/local/lib/ruby/2.7.0/irb/init.rb:18:in `setup'
	1: from /usr/local/lib/ruby/2.7.0/irb/init.rb:121:in `init_error'
/usr/local/lib/ruby/2.7.0/irb/locale.rb:121:in `load': No such file to load -- irb/error.rb (LoadError)
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.13
irb(main):001:0>
cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker run -it ruby:3.0.2-alpine3.14
/usr/local/lib/ruby/3.0.0/irb/locale.rb:121:in `load': No such file to load -- irb/error.rb (LoadError)
	from /usr/local/lib/ruby/3.0.0/irb/init.rb:195:in `init_error'
	from /usr/local/lib/ruby/3.0.0/irb/init.rb:18:in `setup'
	from /usr/local/lib/ruby/3.0.0/irb.rb:402:in `start'
	from /usr/local/lib/ruby/gems/3.0.0/gems/irb-1.3.5/exe/irb:11:in `<top (required)>'
	from /usr/local/bin/irb:23:in `load'
	from /usr/local/bin/irb:23:in `<main>'

https://alpinelinux.org/posts/Alpine-3.14.0-released.html mentions:

The faccessat2 syscall has been enabled in musl. This can result in issues on docker hosts with older versions of docker (<20.10.0) and libseccomp (<2.4.4), which blocks this syscall.

Our version:

cos@runner-8a6210b8-gsrm-1625808786-f58c5cbb ~ $ docker --version
Docker version 19.03.9, build 9d98839

Can we rollback the Alpine v3.14 upgrade and make this opt-in instead of the default while we work to upgrade our systems?

I should note this problem did not occur on Google’s supported cos-85-13310-1260-2 image:

image

Related links:

  1. https://bugzilla.redhat.com/show_bug.cgi?id=1900021
  2. Docker fix: https://github.com/moby/moby/pull/41353
  3. Backport to Docker 19.03.16: https://github.com/moby/moby/pull/41381

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 15
  • Comments: 21 (4 by maintainers)

Commits related to this issue

Most upvoted comments

The same problem with alpine 3.14 on Fargate. @jamesbrooks If you find a way to use the 3.14 on Fargate, please let us know.

please see the alpine linux 3.14 release notes: https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2. you should always read the release notes when upgrading linux distro, or expect surprises.

if, after reading the release notes, it is still not working for anybody, please post:

  1. docker version (not docker --version) and docker info. if you are using hosted Docker, then your provider name and relevant information (e.g. advertised Linux distro and version).
  2. full docker invocation, input (e.g. Dockerfile or stdin), and output

to avoid cluttering the issue, please use <details> element for big output, e.g. https://gist.github.com/ericclemmons/b146fe5da72ca1f706b2ef72a20ac39d.

@alexbrebu see above, and if you are using docker desktop, what version?

@stanhu, docker 19.03 is officially EOL. however, you can continue using it (at your own risk) if you upgrade to runc v1.0.0-rc93 or apply the documented workaround. or, if you don’t care about^W^W^Wdon’t need security at the OS level, which is the only decent reason to be running without ten months of security updates, then just use --security-opt=seccomp=unconfined?

although i don’t maintain this image, i don’t think it is a good idea to downgrade the version. it would be unexpected for users to suddenly get a lower version, as well as a version not matching the latest version advertised on the alpine linux website. additionally, some people may be using the community repo, which is not supported beyond the latest alpine linux release.

From another alpine-bug issue https://github.com/docker-library/php/issues/1177#issuecomment-872475771

The more specific aliases exist specifically to give users an escape hatch to stay on a specific underlying distribution release and control their own consumption and upgrade timing. Just like users using :latest, users using a bare alpine tag should expect some degree of breakage over time, and I don’t think there’s much we can reasonably do about that.

(This is also the common pattern of usage/tagging across all the official images – PHP is not unique in this regard.)

I think the only thing unique about this particular bump is that the most common breakage is more disruptive than usual, but there’s not really much we can do about that, given it’s a change in musl/Alpine itself (https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.14.0#faccessat2).

The same problem with alpine 3.14 on Fargate. @jamesbrooks If you find a way to use the 3.14 on Fargate, please let us know.

This problem happens to me on AWS Fargate (1.4) with ruby:2.7-alpine (1.14). It works when downgraded to ruby:2.7-alpine3.13.

@morgoth @samnang > seems to be working on Fargate 1.4.0 now, tested with ruby:alpine

This problem happens to me on AWS Fargate (1.4) with ruby:2.7-alpine (1.14). It works when downgraded to ruby:2.7-alpine3.13.

On Debian (and derivatives like Raspbian, Ubuntu, etc) the package name is libseccomp2.