moby: download-frozen-image-v2.sh (branch 19.03/master) does not download from non-Docker registries

Description When trying to download container images from non-docker registries (k8s.gcr.io, quay.io, registry.centos.org), download-froxen-image-v2.sh returns “curl: (22) The requested URL returned error: 401 Unauthorized”

Steps to reproduce the issue:

wget https://github.com/moby/moby/blob/19.03/contrib/download-frozen-image-v2.sh
chmod +x download-frozen-image-v2.sh
./download-frozen-image-v2.sh "kube-apiserver" k8s.gcr.io/kube-apiserver:v1.18.2

Describe the results you received: curl: (22) The requested URL returned error: 401 Unauthorized

Describe the results you expected: I expected successful completion of the script with the output looking similar to:

Downloading 'k8s.gcr.io/kube-apiserver:v1.18.2' (1 layers)...

######################################################################## 100.0%

Download of images into 'kube-apiserver' complete.
Use something like the following to load the result into a Docker daemon:
  tar -cC 'kube-apiserver' . | docker load

Additional information you deem important (e.g. issue happens only occasionally): I am able to successfully pull images with download-frozen-image-v2.sh from Docker Hub

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

lient:
 Debug Mode: false

Server:
 Containers: 17
  Running: 17
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-957.27.2.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 3
 Total Memory: 7.638GiB
 Name: 1588216178.test
 ID: ZOYK:SR43:OIMO:ZT36:WBGK:ALKP:WDHV:4NYF:QPN5:GLKF:ZFPB:TPFY
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.): Docker is not installed on the device that I am using download-frozen-image-v2.sh on.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 23 (8 by maintainers)

Most upvoted comments

Don’t mind me…

afflom on May 15, 2020

Fine with me

matya on May 15, 2020

Thanks for reporting; these scripts were created for use in our CI, so definitely have not been designed to be a “universal” solution.

That said, if only minimal changes are needed (and not making the script super-complex to address other uses), I think that would be ok.

Perhaps something like this;

diff --git a/contrib/download-frozen-image-v2.sh b/contrib/download-frozen-image-v2.sh
index 283a9ab34d..ef925b9008 100755
--- a/contrib/download-frozen-image-v2.sh
+++ b/contrib/download-frozen-image-v2.sh
@@ -43,9 +43,9 @@ if [ "$(go env GOHOSTOS)" = 'windows' ]; then
        fi
 fi

-registryBase='https://registry-1.docker.io'
-authBase='https://auth.docker.io'
-authService='registry.docker.io'
+registryBase="${registryBase:-'https://registry-1.docker.io'}"
+authBase="${authBase:-'https://auth.docker.io'}"
+authService="${registryBase:-'registry.docker.io'}"

 # https://github.com/moby/moby/issues/33700
 fetch_blob() {

Then the variables can be overridden before the script is called; something like

registryBase='https://registry-1.example.com' authBase=... authService=... ./download-frozen-image-v2.sh ...

@tianon WDYT?

thaJeztah on May 1, 2020

The v1 and v2 there refer to the version of the registry API they interact with, not the version of the script. 😅

tianon on May 15, 2020

What if we instead keep this script specific to the Moby use case and updated it to link to your more advanced version for users who want to use it to do more than what we need here?

tianon on May 15, 2020