moby: Docker daemon won't start if a registry-mirror is configured in /etc/docker/daemon.json

Description

I have a Nexus 3 docker mirror running in a local server, which my machine points to using /etc/docker/daemon.json. I’ve tested it a few times, got it working, all good.

However, I was using a ridiculously old version of docker and decided to update it in order to use multistage builds (the version I was using couldn’t handle the “AS” alias for stages), in fact, the newest I found(stable).

I completely removed docker, but left the daemon.json there, since I would need it anyway. Installed version 17.12.1 using a “.deb” and it immediately showed an error after the installation was done, showing that docker daemon was unable to start.

When trying to manually start it using dockerd, I got the following message:

invalid mirror: path, query, or fragment at end of the URI “http://192.168.1.201:8081/repository/docker-public/”

After about 30 minutes of searching, I’ve decided to try a different version, just to see what happens, and after downloading version 17.09.1, it started working again

Steps to reproduce the issue:

  1. Create /etc/docker/daemon.json and insert a registry-mirror, like the following:
{
        "registry-mirrors": ["http://192.168.1.201:8081/repository/docker-public/"]
}
  1. Update docker to version 17.12.1 using this link

Describe the results you received: Error when starting docker daemon

Describe the results you expected: Docker daemon should’ve started as usual

Additional information you deem important (e.g. issue happens only occasionally): Also happens with 17.12.0

Output of docker version:

Client:
 Version:	17.12.1-ce
 API version:	1.35
 Go version:	go1.9.4
 Git commit:	7390fc6
 Built:	Tue Feb 27 22:17:40 2018
 OS/Arch:	linux/amd64

Output of docker info:

Can't print, since daemon won't start.

Additional environment details (AWS, VirtualBox, physical, etc.): Using a regular ubuntu 16.04 installation behind a corporate firewall (usually a problem, but not this time I guess)

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 46
  • Comments: 25 (2 by maintainers)

Commits related to this issue

Most upvoted comments

I’ve found a solution ( cat <<EOF [Service] ExecStart= ExecStart=/usr/bin/dockerd -H fd:// --registry-mirror http://$PULL_THROUGT_DN:$REPO_PORT --insecure-registry $PULL_THROUGT_DN:$REPO_PORT --insecure-registry=$SECURE_REGISTRY_DN --insecure-registry=$SECURE_REGISTRY_DN EOF ) | sudo tee /etc/systemd/system/docker.service.d/docker-options.conf and systemctl daemon-reload && systemctl restart docker worked with ubuntu

UPD Domain name without anu subpath

+10
FloMko on Nov 11, 2020

This is a hard problem for us (in my company) - Harbor mirror should be set to /hub project. Podman is very easy to configure to fix this, but Docker / Moby fails.

Can a Moby developer to react on this issue ?

+9
metal3d on Apr 19, 2023

@FloMko does that work with a url where the mirror is at a sub path (https://mirror.example.com/docker/mirror)? Or just if it’s a domain name (https://mirror.example.com)?

Only FQDN without any subpath (comment has been updated)

+8
FloMko on Nov 11, 2020

Had a quick chat with some people on our slack, related to https://github.com/moby/moby/pull/38991#discussion_r271517220, because at the time, containerd did not have support for these.

Since then, containerd looks to have added support https://github.com/containerd/containerd/blob/adeba792f16bff6d8f68d560566ed50202634bac/remotes/docker/registry.go#L71, however, it has not yet been looked into if that can be used (and BuildKit would have to be updated to user/support it, as it currently doesn’t)

+4
thaJeztah on Aug 20, 2020

I’m seeing the same behavior with our Nexus docker group (private repo + proxy for public images).

Docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64

Our /etc/docker/daemon.json:

{
  "insecure-registries":[
    "192.168.1.10"
  ],
  "registry-mirrors": [
    "http://192.168.1.10/nexus/repository/docker-group"
  ]
}

FWIW, this only seems to be an issue with Docker. Gems, Apt, YUM, NPM, and Maven can all pull through Nexus just fine.

+4
karlwilbur on Jul 6, 2018

I’m seeing the same behavior with our Nexus docker group (private repo + proxy for public images).

Docker version:

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64

Our /etc/docker/daemon.json:

{
  "insecure-registries":[
    "192.168.1.10"
  ],
  "registry-mirrors": [
    "http://192.168.1.10/nexus/repository/docker-group"
  ]
}

FWIW, this only seems to be an issue with Docker. Gems, Apt, YUM, NPM, and Maven can all pull through Nexus just fine.

For Nexus: follow this post https://mtijhof.wordpress.com/2018/07/23/using-nexus-oss-as-a-proxy-cache-for-docker-images. It allows you to serve the repo at a different port, mitigating the need for a path.

+3
Nebula83 on Dec 23, 2020

Any update on this? Not being able to use Nexus for our mirror is a huge pain

+3
allout58 on Jan 31, 2020

Wild that we all have to set up elaborate reverse-proxy systems to get this to work for private registry mirrors. The last update here from a project maintainer seemed to indicate that this was a limitation years ago… but yet here we are in 2023 still going “I have to do what to make this work?”

+1
kevholmes on Feb 17, 2023

The problem ist the validation function:

https://github.com/moby/moby/blob/8d193d81af9cbbe800475d4bb8c529d67a6d8f14/registry/config.go#L323-L340

uri.Path != "" && uri.Path != "/" does not allow subpaths.

Also to see in unit tests:

https://github.com/moby/moby/blob/8d193d81af9cbbe800475d4bb8c529d67a6d8f14/registry/config_test.go#L131-L160

+1
dklimpel on Jan 9, 2023

I’m observing similar behavior. Docker version:

docker version
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:43 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8891c58
  Built:            Mon Dec 28 16:15:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

This config doesn’t work:

{
  "insecure-registries": ["insecure.server.example.com:4567"],
  "data-root": "/home/docker",
  "max-download-attempts": 65535,
  "registry-mirrors": ["https://artifactory.local.corp.net/artifactory"]
}

Docker daemon complains in logs: failed to start daemon: invalid mirror: path, query, or fragment at end of the URI "https://artifactory.local.corp.net/artifactory"

However, if I move registry-mirrors to separate pair of curly braces, it works

{
  "insecure-registries": ["insecure.server.example.com:4567"],
  "data-root": "/home/docker",
  "max-download-attempts": 65535
},
{
  "registry-mirrors": ["https://artifactory.local.corp.net/artifactory"]
}
+1
wl2776 on Jan 15, 2021

Since it’s a Nexus 3 registry-mirror, and not one that I’ve configured, that path is actually what points which of the repositories is a “docker-proxy”.

Removing it in 17.12.1 resulted in docker ignoring the registry-mirror and trying to fetch an image directly. Same in 17.09.0, however leaving the path there in 17.09.0 makes it work.

+1
mohr023 on Mar 14, 2018
Read more comments on GitHub
← moby: docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
moby: Docker DNS not resolving certain addresses →