moby: Docker pull not resolving private registry address that's been set in the host file

Description

After upgrading to version 23.0+ it seems docker is ignoring the host machines dns entry that’s been set in the host file. Our company uses two proget private docker registries hosted in 2 different servers that has separate ip addresses but sharing the same dns name. The one registry acts as our devtest registry for developers to pull and push packages to it and once sign off of those packages has completed, those packages will then get replicated to the other registry for production use.

We have a set of build agents that are being used by our developers to perform their dev/test work. On these agents we have hardcoded the dns name as well as the ip address of the devtest registry so all pull and push activities within the build agent will be pointing to the devtest registry. Without doing so, it seems by default it will point to the production registry.

This has been working all along using docker engine version 20.10.21.

Now that I’ve upgraded docker engine to 23.0.1, while performing docker pull/push commands it looks like it is not resolving the registry to the ip address that’s been set in the host file anymore.

docker pull company.com/imagebuild/some-internal-image Error I get is

Error response from daemon: unknown: Could not resolve ProGet feed "imagebuild". Docker respository names in ProGet must include the feed as a prefix

The imagebuild feed only exists in our devtest registry so the above looks to me like docker is somehow not resolving the dns name from the host file.

I’ve also upgraded the engine to 23.0.2 recently and it seems it’s behaving the same.

Is there any changes to the internals in terms of networking between 20.10+ and 23.0+?

Reproduce

Install/Upgrade to docker 23.0.1 or 23.0.2
On host machine add entry to host file with ip address and dns name of the private registry
Try perform a docker pull of an image that exists in the private registry by using the dns name of the registry

Expected behavior

docker pull should be able to pull the image from the private registry

docker version

Client:
 Version:           23.0.2
 API version:       1.42
 Go version:        go1.19.7
 Git commit:        569dd73
 Built:             Mon Mar 27 16:18:04 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.2
  API version:      1.42 (minimum version 1.24)
  Go version:       go1.19.7
  Git commit:       219f21b
  Built:            Mon Mar 27 16:15:45 2023
  OS/Arch:          windows/amd64
  Experimental:     false

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.2
    Path:     C:\ProgramData\Docker\cli-plugins\docker-compose.exe

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 6
 Server Version: 23.0.2
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 20348 (20348.1.amd64fre.fe_release.210507-1500)
 Operating System: Microsoft Windows Server Version 21H2 (OS Build 20348.1607)
 OSType: windows
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7GiB
 Name: alanc-win000000
 ID: f6da5401-c9cf-4493-8aef-52f11d38b2a6
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: true
  File Descriptors: -1
  Goroutines: 24
  System Time: 2023-03-31T21:18:51.1932554Z
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional Info

Note that this is related to docker for windows, installed on windows server 2022

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (11 by maintainers)

Most upvoted comments

@iZnoGod722 we merged fixes in our build scripts which should fix this issue (those changes will be used for upcoming patch releases).

Thank you SO MUCH for narrowing it down to that specific patch release (leading me to the Go update). This definitely was a fun one (as most things “continued to work as usual”). I also raised this issue to other teams at Docker, and will be giving some other projects a heads-up, as it’s not unlikely there’s other code bases being affected as well.

So the issue is due to “file based resolution” (read: etc/hosts) just … not being implemented yet in Go 1.19’s native resolver. And a TODO (basically; “does windows support etc/host? Let’s figure that out later”) was added instead.

// TODO(bradfitz): implement files-based
// lookup on Windows too? I guess /etc/hosts
// kinda exists on Windows. But for now, only
// do DNS.

That omission was not (at least not clearly) mentioned in the release notes (but someone reported an issue about it; https://github.com/golang/go/issues/57757)

Fun fact: as localhost is resolved using etc/hosts, it also means that localhost won’t resolve, so it definitely is severely broken (more so combining that with a previously “no-op” buildtag now becoming active and introducing the breaking change).

thaJeztah on May 24, 2023