moby: Docker is not passing auth informations when pulling from a mirror registry

@thaJeztah Could you please explain:

If a mirror is configured, and that mirror itself requires authorisation, the client should be authenticated against that mirror (in which case those credentials would be used)

If I understand correctly this is exactly what isn’t working, and what started this whole issue.

docker login my-registry # my-registry is configured as the mirror

docker pull my-registry/my-image # works
docker pull my-image # fails

Am I missing something? Is there any way to pull from a password protected mirror without reverse-proxying it with an un-authenticated server?

any workaround in 2020? thanks

please don’t spam useless +1 comments. they only annoy everyone subscribed to this issue, the majority being people who are looking for a resolution. this brings no benefit to anyone. if you want to add an indicator that you’re also affected by this you can add a reaction to the first post.

@thaJeztah

doing so would be be a security issue as it would leak credentials to any registry that’s configured as mirror (which should have no access to them).

This is exactly what is happening though; Docker Hub credentials are being sent to the mirror. These are my logs after logging in to Docker Hub and trying to execute docker pull mysql:5.7:

level=debug msg="Calling POST /v1.41/auth"
level=debug msg="form data: {\"password\":\"*****\",\"serveraddress\":\"https://index.docker.io/v1/\",\"username\":\"MY_USER_NAME\"}"
level=debug msg="hostDir: /etc/docker/certs.d/MY_MIRROR"
level=debug msg="attempting v2 login to registry endpoint https://registry-1.docker.io/v2/"
level=debug msg="Calling HEAD /_ping"
level=debug msg="Calling HEAD /_ping"
level=debug msg="Calling HEAD /_ping"
level=debug msg="Calling GET /v1.41/info"
level=debug msg="Calling POST /v1.41/images/create?fromImage=mysql&tag=5.7"
level=debug msg="hostDir: /etc/docker/certs.d/MY_MIRROR"
level=debug msg="Trying to pull mysql from https://MY_MIRROR/v2"
level=info msg="Attempting next endpoint for pull after error: Head https://MY_MIRROR/v2/library/mysql/manifests/5.7: unknown: Bad credentials"

It logs in to Docker Hub and retrieves an access token, then tries to use that token with the mirror.

If a mirror is configured, and that mirror itself requires authorisation, the client should be authenticated against that mirror (in which case those credentials would be used)

I think we all agree this is what should happen; however, it is not happening. Instead, Docker Hub credentials are being sent, which, as you note, is a big security issue.

This is still an issue for me

This is an issue for me as well. I need to manage a registry-mirror between different sites and so the reverse-proxy must be open to the internet. I had to manually restrict the IP addresses that could access the registry since the only solution is to add the credentials to the reverse-proxy. Leaking credentials is not an option.

A solution for this problem is long overdue. Using the credentials set by docker login for the configured registry-mirror when trying to pull images only makes sense to me.

@LiohMoeller yeah you have to use the same username and password on both your registry and Docker Hub for this workaround to work fully.

It’s absolutely insane that this issue is still open.

There is a “workaround” for this. It really is silly though and I don’t recommend actually doing this. It also relies on the credentials coming from the docker client, not dockerd.

It (ab)uses the fact that docker sends the docker hub credentials to the mirror.

🚨 Only do this if you don’t care about leaking your credentials to Docker Hub!

/etc/docker/daemon.json:

{
  "registry-mirrors": ["https://docker.yourmirror.com"]
}

docker login -u mirroruser docker.yourmirror.com

Then edit ~.docker/config.json and duplicate the docker.yourmirror.com entry for https://index.docker.io/v1/:

{
        "auths": {
                "docker.yourmirror.com": {
                        "auth": "bWlycm9ydXNlc..."
                },
                "https://index.docker.io/v1/": {
                        "auth": "bWlycm9ydXNlc..."
                }
        }
}

If you use a credsStore, manage the equivalent entry in your credentials store.

---

Docker will send the configured credentials for https://index.docker.io/v1/ to your configured registry mirror. You won’t be able to use the official registry (without changing your configuration).

Again, this will send your credentials to docker.io when the mirror is not used. If you’re okay with that, it might be hack that works for you.

This behaviour is not a bug, as authorization / credentials are tied to a host, and should not be sent to a different host (similar to when a redirect is performed, credentials should not be forwarded to the host that’s redirected to; doing so would be be a security issue as it would leak credentials to any registry that’s configured as mirror (which should have no access to them).

If a mirror is configured, and that mirror itself requires authorisation, the client should be authenticated against that mirror (in which case those credentials would be used)

Nexus OSS 3.6.0-02 can finally transparently proxy docker images. It has a new feature called “Anonymous Read Access” for docker registry access (see https://help.sonatype.com/display/NXRM3/Private+Registry+for+Docker) and disabling “Force basic authentication” and adding “Docker bearer token realm” in nexus/admin/security/realms seems to fixes this issue, no more “no basic auth credentials” in the logfile.

Same issue here. Using Docker 17.06.2-ce and Artifactory 5.4.6 as a registry mirror. This is actually pretty blocking for my organization because our Docker server does not have internet access and our Artifactory has authentication. Is there a workaround available? I know about setting the request header in the reverse proxy but this only works for pulling.

+1

@vdotjansen and at present this is a 3 year old bug with no workaround short of running a local proxy server that passes the credentials? I think this is a more pressing problem in that Docker Hub is putting in those usage limits.

The problem gets bigger for us as we are going to need to pull docker images from outside our organization we need to be sure that it is only done by people we trust and therefor we need to add authentication and authorization, how can we do this?

This is still an issue for me

if you use nexus3 (3.29.1-01) and still doesn’t work, try below process

1. nexus configuration -> system -> capabilities -> create capability -> default role -> check enable this capability and select nx-anonymous

2. add configuration /etc/docker/daemon.json (centos7) { “registry-mirros” : “your nexus docker proxy url” (ex: https://my.registry.com:5000) }

3. systemctl restart docker

when i tried no.2 process without no.1, it doesn’t work. after i add no.1 process, registry mirrors works!

Having the same issue, where it fails the pull even though it is (pre)authenticated against the mirror (and not the upstream)

docker login SOME_MIRROR 
docker pull SOME_MIRROR/some-image # works
docker pull some-image # fails

Attempting next endpoint for pull after error: Get https://SOME_MIRROR/v2/library/nginx/manifests/1.19.5: no basic auth credentials

~/.docker/config.json

{
        "auths": {
                "SOME_MIRROR": {
                        "auth": "<redacted>"
                },
        }
}

docker version: 18.09.8

---

Also:

• buildkit on the other hand uses the auth correctly, e.g. DOCKER_BUILDKIT=1 docker build ... with a FROM some-image would correctly pull this from the mirror with the auth.

Wow… 3 years and no resolution yet ? this hurts my heart…

This is still an issue for me.

The thing is I was authorized against the mirror. XXXX is the one in the registry mirrors. YYYY is my repo itself XXXX and YYYY point to the same server just have different DNS names because I was trying to debug the problem.

$ cat config.json | jq
{
  "HttpHeaders": {
    "User-Agent": "Docker-Client/19.03.12 (windows)"
  },
  "auths": {
    "XXXX": {},
    "https://index.docker.io/v1/": {},
    "YYYY": {}
  },
  "credStore": "desktop",
  "credsStore": "desktop",
  "stackOrchestrator": "swarm"
}

@marcelmaatkamp We cannot remove the auth for our Nexus instance (as you described) is there a possibility for adding login credentials for the dockerd in some way? I even tries adding user:pass to the mirror url. Does not work either.

As @TristanCP said in stackoverflow, the workaround helps.

added in nginx docker config:

proxy_set_header Authorization "Basic a2luZzppc25ha2Vk";

Example:

server {
	listen *:443 ssl http2;
	server_name docker.domain.blah.net;
	ssl on;
	include ssl/domain.blah.net.conf;
	# allow large uploads of files - refer to nginx documentation
	client_max_body_size 0;
	chunked_transfer_encoding on;
	location / {
		proxy_pass http://127.0.0.1:8083/;
		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header Authorization “Basic YWRtaW46YWRtaW4xMjM=”;

		#proxy_set_header X-Forwarded-Proto "https";
	}
}

server {
	listen *:80;
	server_name docker.domain.blah.net;
	return 301 https://$server_name$request_uri;
}

Docker tries to authenticate to your mirror with the login credentials for Docker Hub. If you create a user in your mirror with the same username and password as your Docker Hub account, the mirroring will start to work.

I had the same issue. I set up a Sonatype Nexus instance as Docker Hub mirror, hosted at registry.example.com. Nexus requires authentication (anonymous mode disabled). My Docker host is authenticated to Docker Hub as user A, and to Nexus as user B.

If I pull registry.example.com/mygroup/myservice:latest Docker uses the user B credentials as expected. If I pull nginx:latest Docker tries to get it from the mirror (Nexus) using the Docker Hub credentials (user A) to authenticate, which fails. Thus it falls back to index.docker.io. After adding a new user in Nexus with user A’s credentials, pulling nginx:latest does work through the mirror as expected. In Nexus I can also see the cached nginx version.

So an ugly workaround is to add all Docker Hub credentials to your Mirror. In our case that is acceptable for our infrastructure servers that use a single service user account, but we can’t add all Docker Hub accounts of our users to our Nexus…

Feels like the issue somehow related to that docker thinks that shell is not interactive when you are working over ssh. I had this error “no basic auth credentials” when I was connected over ssh, after I connected over VNC and opened terminal on remote machine - everything worked,

5 years later, Docker version 24.0.1, build 6802122, I am afraid I stumbled upon same issue. In daemon.json: repository-mirrors: ["http://my.mirror:5001"], insecure-registries: ["my.mirror:5001"]

docker login http://my.mirror:5001 (success) docker pull my.mirror:5001/postgres:14.2 - works fine. docker pull postgres:14.2 - fails with:

level=info msg="Attempting next endpoint for pull after error: Head \"http://my.mirror:5001/v2/library/postgres/manifests/14.2\": no basic auth credentials" and then continues to try downloading from repository-1.docker.io (and fails on timeout due to firewalls)

Anyone got this working by any chance?

+1

Are there some PR trying to address this ?

Another workaround is to use separate client configs based on where you need to authenticate - see https://github.com/moby/moby/issues/42022#issuecomment-984617195

This is still an issue for me

This problem prevents the GCR mirror from working if you are authenticated with Docker Hub, as it tries to authenticate on GCR with Docker Hub credentials and fail.

time="2021-02-12T23:50:30.245560572Z" level=info msg="Attempting next endpoint for pull after error: Head https://mirror.gcr.io/v2/confluentinc/cp-zookeeper/manifests/5.3.0: unauthorized: Not Authorized."
time="2021-02-12T23:50:38.451982466Z" level=info msg="Attempting next endpoint for pull after error: Head https://mirror.gcr.io/v2/confluentinc/cp-enterprise-kafka/manifests/5.3.0: unauthorized: Not Authorized."
time="2021-02-12T23:50:40.310703679Z" level=info msg="Attempting next endpoint for pull after error: Head https://mirror.gcr.io/v2/confluentinc/cp-schema-registry/manifests/5.3.0: unauthorized: Not Authorized."
time="2021-02-12T23:50:49.609721932Z" level=info msg="Attempting next endpoint for pull after error: Head https://mirror.gcr.io/v2/library/postgres/manifests/11-alpine: unauthorized: Not Authorized."
time="2021-02-12T23:50:52.051122780Z" level=info msg="Attempting next endpoint for pull after error: Head https://mirror.gcr.io/v2/library/redis/manifests/5.0.6-alpine: unauthorized: Not Authorized."
time="2021-02-12T23:50:53.036023645Z" level=info msg="Attempting next endpoint for pull after error: Head https://mirror.gcr.io/v2/scylladb/scylla/manifests/4.2.0: unauthorized: Not Authorized."

@jhkim8406 so basically you don’t have any authentication then. Not really what we are looking for unless the private mirror is also firewalled to prevent other people from accessing.

I’ve just noticed this issue when migrating a Nexus3 instance & was wondering why the docker mirror wasn’t being used.

One thing I can add here is that, for me, it’s normal users that are affected when pulling an image. When root does the pull it does go via the proxy as expected.

This is on Ubuntu 20.04 & Docker 19.03.8

We use workaround as proposed by JOMO above and we prevent leaking credentials to Docker Hub to redirect all docker hub to localhost example to set in file /etc/hosts where the docker engine is running

127.0.0.1   index.docker.io
127.0.0.1   registry-1.docker.io
127.0.0.1   docker.io
...

Hello all!

Same issue…

I’m Using Sonatype Nexus 3 as to proxy registry-1.docker.io and act as a mirror. HTTPS and nginx configured properly (docker login successful)

Works fine: docker pull docker.domain.blah.net/rancher/server

Don’t work: docker pull rancher/server

Docker version 1.13.1, build 092cba372

cat  ~/.docker/config.json 
{
	"auths": {
		"127.0.0.1:8083": {
			"auth": "YWRtaW46YWRtaW4xMj1="
		},
		"docker.domain.blah.net": {
			"auth": "YWRtaW46YWRtaW4xMj1="
		}
	}
}

DEBU[0013] Calling GET /_ping                           
DEBU[0013] Calling POST /v1.26/containers/create?name=docker-rancher-server 
DEBU[0013] form data: {"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":[],"ExposedPorts":{"8080/tcp":{}},"HostConfig":{"AutoRemove":false,"Binds":["/var/run/docker.sock:/var/run/docker.sock","/home/phl/devops/rancher/mysql:/var/lib/mysql","/etc/ssl/localcerts/domain.blah.net.crt:/var/lib/rancher/etc/ssl/ca.crt"],"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":null,"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{"8080/tcp":[{"HostIp":"","HostPort":"8180"}]},"Privileged":true,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"unless-stopped"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"rancher/server","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""} 
ERRO[0013] Handler for POST /v1.26/containers/create returned error: No such image: rancher/server:latest 
DEBU[0013] Calling GET /v1.26/info                      
DEBU[0013] Calling POST /v1.26/images/create?fromImage=rancher%2Fserver&tag=latest 
DEBU[0013] hostDir: /etc/docker/certs.d/docker.domain.blah.net 
DEBU[0013] Trying to pull rancher/server from https://docker.domain.blah.net/ v2 
ERRO[0013] Attempting next endpoint for pull after error: 

Get https://docker.domain.blah.net/v2/rancher/server/manifests/latest: 

no basic auth credentials

DEBU[0013] Trying to pull rancher/server from https://registry-1.docker.io v2 
DEBU[0025] Pulling ref from V2 registry: rancher/server:latest 
DEBU[0025] pulling blob "sha256:6599cadaf950a71af51aa84b85e6cdb1990287b79f8e71a78f6986b318d4c4a0"