moby: manifest list (multiarch) picks wrong arch on ARMv6

Description

The command docker pull executed on a armv6 Raspberry Pi One (“Model B Plus Rev 1.2”) running a 32-bit Raspbian OS will fail to pull the correct docker image defined in a manifest as soon as a ‘linux/arm/v7’ manifest entry exists.

Steps to reproduce the issue:

1. uname -m && docker run --rm arm32v6/alpine:3.8 sh -c 'cat /etc/*release | grep PRETTY_NAME'
2. uname -m && docker run --rm arm32v6/alpine:3.9 sh -c 'cat /etc/*release | grep PRETTY_NAME'
3. uname -m && docker run --rm alpine:3.8 sh -c 'cat /etc/*release | grep PRETTY_NAME'
4. uname -m && docker run --rm alpine:3.9 sh -c 'cat /etc/*release | grep PRETTY_NAME'

Describe the results you received:

$ uname -m && docker run --rm arm32v6/alpine:3.8 sh -c 'cat /etc/*release | grep PRETTY_NAME'
armv6l
PRETTY_NAME="Alpine Linux v3.8"
$ uname -m && docker run --rm arm32v6/alpine:3.9 sh -c 'cat /etc/*release | grep PRETTY_NAME'
armv6l
PRETTY_NAME="Alpine Linux v3.9"
$ uname -m && docker run --rm alpine:3.8 sh -c 'cat /etc/*release | grep PRETTY_NAME'
armv6l
PRETTY_NAME="Alpine Linux v3.8"
$ uname -m && docker run --rm alpine:3.9 sh -c 'cat /etc/*release | grep PRETTY_NAME'
armv6l
$ echo $?
139

For steps 1,2 and 3, I was expecting these results. However, at step 4, the ‘docker run’ command did not produce any output and exits with a 139 return code.

Describe the results you expected:

For step 4, I was expecting the same result as in step 2:

armv6l
PRETTY_NAME="Alpine Linux v3.9"

Additional information you deem important (e.g. issue happens only occasionally):

There is no “linux/arm/v7” image on alpine:3.8 manifest:

$ docker run --rm mplatform/mquery alpine:3.8Image: alpine:3.8
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm/v6
   - linux/arm64
   - linux/386
   - linux/ppc64le
   - linux/s390x

But alpine:3.9 manifest defines a linux/arm/v7 entry:

$ docker run --rm mplatform/mquery alpine:3.9
Image: alpine:3.9
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm/v6
   - linux/arm/v7
   - linux/arm64
   - linux/386
   - linux/ppc64le
   - linux/s390x

If you prefer, with docker manifest inspect:

$ DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect alpine:3.8
[...]
"digest": "sha256:dabea2944dcc2b86482b4f0b0fb62da80e0673e900c46c0e03b45919881a5d84",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
[...]

$ DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect alpine:3.9
[...]
{
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 528,
         "digest": "sha256:7a3d88cbc7e2d6c0213deaf2d006933c9f5905c4eb7846b703a66fc6504000b7",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 528,
         "digest": "sha256:cfd8b55d209956f63c8fcc931f5c6874984e5e0ffdcb8f45ba9085f190385d73",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
[...]

An evidence that dockerd 19.03.8 pulls the (wrong) linux/arm/v7 images variant:

$ docker pull arm32v7/alpine:3.9
$ docker images | grep "alpine"
arm32v6/alpine        3.8                                                   e60f4bcd6c89        4 months ago        4.01MB    # OK
alpine                3.8                                                   e60f4bcd6c89        4 months ago        4.01MB    # OK
arm32v6/alpine        3.9                                                   4fe9c9b2a0c4        4 weeks ago         4.68MB    # OK
arm32v7/alpine        3.9                                                   9df0ff5446fc        4 weeks ago         3.72MB    # OK
alpine                3.9                                                   9df0ff5446fc        4 weeks ago         3.72MB    # NOK: should be '4fe9c9b2a0c4' and not '9df0ff5446fc' ! 

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:35:24 2020
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:22 2020
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 34
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 2zsjxcgeymlaecu00y8j0i1uw
  Is Manager: true
  ClusterID: t5ftctfw0dlnxbve8ilm7yo0x
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 192.168.0.114
  Manager Addresses:
   192.168.0.114:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.97+
 Operating System: Raspbian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: armv6l
 CPUs: 1
 Total Memory: 432.4MiB
 Name: white
 ID: QDNQ:Y3WR:ALAX:BXDG:JRKM:VBXW:SEC3:LX7D:LLFH:GG2O:EHK6:GG4G
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpuset support

Additional environment details (AWS, VirtualBox, physical, etc.):

This test was ran on a “Raspberry Pi Model B Plus Rev 1.2”. Important device info: it is an armhf v6 (and not v7 !) device, even if its ‘CPU architecture’ is set to 7 !

$ dpkg --print-architecture
armhf
$ uname -m
armv6l
$ cat /proc/cpuinfo | grep "CPU architecture"
CPU architecture: 7