moby: manifest list (multiarch) picks wrong arch on ARMv7
Description
With the “new” manifest list for multi-arch support, there is a corner case on ARM where the picked-up architecture is not ideal. On a ARMv7 CPU running Docker (e.g. a Raspberry Pi 2/3), the architecture is armhf (so with floating point hardware support). However when pulling an image such as the official Debian, the “armel” variant is pulled. This is compatible with ARMv7 but emulates floating point operations instead of using the underlying hardware.
Steps to reproduce the issue:
- on a ARMv7 SoC (e.g Raspberry Pi 2 or 3), install Linux + Docker CE
- do
docker run --rm -it debian dpkg --print-architecture
Describe the results you received:
It displays armel
. Meaning it has pulled the suboptimal image.
Describe the results you expected:
It should display armhf
. Meaning it has pulled the adequate image.
Additional information you deem important (e.g. issue happens only occasionally):
See @tianon tweet (and the rest of the thread for more info): https://twitter.com/tianon/status/909084978515927040
Output of docker version
:
Client:
Version: 17.06.2-ce
API version: 1.30
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 20:08:20 2017
OS/Arch: linux/arm
Server:
Version: 17.06.2-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 20:01:53 2017
OS/Arch: linux/arm
Experimental: false
Output of docker info
:
Containers: 7
Running: 5
Paused: 0
Stopped: 2
Images: 157
Server Version: 17.06.2-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.12.10-v7-lowlat-rtc1307+
Operating System: Raspbian GNU/Linux 9 (stretch)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 968.7MiB
Name: pi2-01.lan.berthon.eu
ID: (redacted)
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
n/a
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 13
- Comments: 21 (16 by maintainers)
Commits related to this issue
- Update README to mention manifest lists — committed to docker-library/official-images by tianon 7 years ago
- Moby/moby: Add variant support in download-frozen-image-v2.sh Since for now download-frozen-image-v2.sh won't be smart enough to use the adequate image in arm platform, hence we add variant support t... — committed to Pennyzct/moby by Pennyzct 6 years ago
- distribution: Add arch variant matching functionality for Arm When pulling multi-arch image from official repo, mismatched arch variant may be pulled,because only arch and os will be verified current... — committed to Pennyzct/moby by Pennyzct 6 years ago
- cicd: build-agent: Fix Docker manifest variant for ARMv6 We need to use adjust the variant for Docker manifest proper use. This can be seen at: https://github.com/moby/moby/issues/34875#issuecomment... — committed to shellhub-io/shellhub by otavio 4 years ago
- cicd: build-agent: Fix Docker manifest variant for ARMv6 We need to use adjust the variant for Docker manifest proper use. This can be seen at: https://github.com/moby/moby/issues/34875#issuecomment... — committed to shellhub-io/shellhub by otavio 4 years ago
wow, I recal annoying Phil and tianon about this @dockerCon in 2017, and today, it hits me on a hobby project.
memories
It’s definitely not anywhere near ready for use, but here’s what I’ve got so far in my own hacking (which may or may not be useful for elsewhere, and needs to live elsewhere since so many other places need to use this same code/logic):
I’ve successfully verified that I can pull
i386/hello-world:latest
onamd64
with this approach (which currently fails due to it being a386
-only manifest list), although I getError response from daemon: oci runtime error: target os mismatch with current os linux.
when I try to run it. 😅Also, I’m really disappointed to note that
runtime.GOARM
doesn’t exist –runtime.goarm
does, but would require some hanky cgo in order to access due to it being private, so we’re likely going to be better off doing runtime detection in the case ofGOARCH
ofarm
orarm64
.I think https://github.com/golang/go/blob/718d9de60fd4337d9044cdc2c685177dd2177ef6/src/runtime/os_linux_arm.go is probably a useful reference, if we can successfully recreate the value of
hwcap
(which appears to come from/proc/self/auxv
) to test against it. Basically, ifhwcap&_HWCAP_VFP == 0
, we have no floating point unit and should preferv5
only, else ifhwcap&_HWCAP_VFPv3 == 0
, we preferv6
followed byv5
, and otherwise, we preferv7
thenv6
thenv5
.@yosifkit - I did a few tests with a raspberry pi 1 (not a zero) with docker 19.03.8 about your ‘hack’, and I think it “looks to work” only if you amend both v6 and v7 variants with the additional ‘l’ (as you did on https://github.com/KEINOS/Dockerfile_of_Alpine/issues/3). But I have unfortunately the feeling that it only “looks to work”, but actually, it does not. You could get an idea of the tests I did by analyzing the public manifests “biarms/mysql:test-moby-issue-34875-tc1” to “biarms/mysql:test-moby-issue-34875-tc4” (with command line
DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect biarms/mysql:test-moby-issue-34875-tc1
) and test them with commands likedocker run --rm mplatform/mquery biarms/mysql:test-moby-issue-34875-tc1
anddocker run -it --rm biarms/mysql:test-moby-issue-34875-tc3 --version
to get an idea. FYI, to perform my tests, I also set the log level to trace for my docker service with:And here are my conclusions:
The “pb” with your hack is that a real arm v7 device (in my case, an odroid device also running docker 19.03.8) will also get the v6 image. So yes, it will work, as arm-v7 is backward compatible with arm-v6. But it will not work as expected. If the goal is only to have something that ‘run’, then only publish the arm v6 and never the arm v7, and it will also solve the pb (without v6l hack).
My tests on a rpi1:
By the way, aarch64, there is no pb :
I realize that ARM hardware “variant inspection” is the hard problem being danced around (and @stevvooe and I just had a talk with the ARM, Ltd. team lead at the Moby Summit last week on this topic), but that is the correct solution, as it corrects the
TODO
around our code completely ignoring variant at this point. ThisTODO
has been sitting in the code since Docker 1.10 or so now: https://github.com/moby/moby/blob/master/distribution/pull_v2.go#L718-L726Ordering is, as noted above, only a workaround and can only be “depended” on for certain tools that we have control over (like the Docker engine or containerd). Given the spec speaks nothing of ordering, then it is not a good solution for verifying that others will follow some general ordering for “best case” scenarios.
@tianon might be able to fix this by changing the order in which images are listed in the manifest. For debian image, we have the following:
Note that
v5
is listed beforev7
. Based on the current matching, it will favor thev5
image because it doesn’t consider the variant.@yosifkit
Indeed, we should not. I agree.
Since it should work with both
v6
andv7
and it doesn’t, for almost 3 years. I understand that it’s more to the ARM problem rather than Docker’s multi-arch detection functionality.I just needed a stable(?) workaround to make a single Dockerfile for both architectures until this issue gets solved. But I should’ve mentioned that. Thanks, I will add a note to my post.
A variant value of
v6l
is not part of the OCI image specification, so it should not be used.v6
is the proper value for all Armv6 devices.F.Y.I.
Hi, here’s a little hack/tip for those who want to create a single but multi-arch manifest list that is compatible with both ARMv6 and ARMv7 (such as RaspberryPi ZeroW and RPi3B+).
TL; DR
variant
value of the manifest fromv6
tov6l
andv7
tov7l
for each architecture in the manifest list,TS; DR
This image should work on: RPi Zero(ARMv6), RPi 3B+(ARMv7), macOS(Intel), Win10(Intel), and Linux(Intel) machines.
v6l
is not part of the OCI image specification. Since it aims to work with thev6
variant and to avoid confusion in the future, one should usev6
rather thanv6l
. So keep in mind that this is a temporary workaround until this issue gets solved.From reading through the issue here I am unsure whether something is done about it.
On my Raspberry Pi 3+ my Docker reports:
For example I can run Alpine directly:
However, Python within the same Alpine release:
My guess is that the manifests of these library images are not equal. Is there anyway I can force this locally?
Looks like there’s others running into that; https://gist.github.com/lucab/f7162ca2d95191c692edc12ea8ccaaef, and found some discussion in https://github.com/golang/go/issues/9737 (and https://github.com/golang/go/commit/1b53f15ebb00dd158af674df410c7941abb2b933)
Should we open an issue with Golang to make it public?
@ijc The current convention and behavior of docker is to take the first match. I agree this needs to be more clever but ordering can be used as a fall back when all else fails.
As of now, I have no clue how to detect variant on a given host. I am trying to get some ARM builders in containerd so we can get this right over there. Ideas are welcome.