moby: docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers

Output of docker version:

Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 53
 Running: 0
 Paused: 0
 Stopped: 53
Images: 331
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 437
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 3.13.0-86-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.66 GiB
Name: 3kolor
ID: BAOT:3C5I:WCTI:RXDF:5W55:BJTA:MRDP:NSAW:4H4R:ILDC:BG3Z:C65X
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): true
 File Descriptors: 12
 Goroutines: 35
 System Time: 2016-05-10T19:54:31.370342285+02:00
 EventsListeners: 0
Username: almoehi
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.): Ubuntu 14.04.4 LTS Linux 3.13.0-86-generic #130-Ubuntu SMP Mon Apr 18 18:27:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce the issue:

  1. docker login

Describe the results you received: It seems like docker daemon cannot access the internet, even though internet access is working on the box. I cant pull images and I cant even log-in to the registry. I’ve re-installed the whole docker packages and rebooted the machine several times - no success.

docker login:

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username (******): 
Password: 
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

manually running a request with CURL works without problems:

 curl https://registry-1.docker.io/v2/
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

running docker pull ubuntu just hangs forever without producing any debug output in the daemon logfile and after couple of minutes dies with following message:

Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"

Describe the results you expected: Docker daemon should be able to access the internet. Docker should successfully login to registry and pull images.

Additional information you deem important (e.g. issue happens only occasionally):

debug logfile

time="2016-05-10T19:56:01.669665420+02:00" level=debug msg="Calling GET /v1.23/info" 
time="2016-05-10T19:56:05.459596278+02:00" level=debug msg="Calling POST /v1.23/auth" 
time="2016-05-10T19:56:05.459709902+02:00" level=debug msg="form data: {\"email\":\"*****\",\"password\":\"*****\",\"serveraddress\":\"https://index.docker.io/v1/\",\"username\":\"******\"}" 
time="2016-05-10T19:56:05.459899278+02:00" level=debug msg="attempting v2 login to registry endpoint https://registry-1.docker.io/v2/" 
time="2016-05-10T19:56:06.429620835+02:00" level=debug msg="Increasing token expiration to: 60 seconds" 
time="2016-05-10T19:56:20.928767529+02:00" level=error msg="Handler for POST /v1.23/auth returned error: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" 

debug logfile when running docker pull:

time="2016-05-10T20:03:52.257721929+02:00" level=error msg="Error trying v2 registry: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: \"<html><body><h1>408 Request Time-out</h1>\\nYour browser didn't send a complete request in time.\\n</body></html>\\n\\n\"" 
time="2016-05-10T20:03:52.257784072+02:00" level=error msg="Attempting next endpoint for pull after error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: \"<html><body><h1>408 Request Time-out</h1>\\nYour browser didn't send a complete request in time.\\n</body></html>\\n\\n\"" 
time="2016-05-10T20:03:52.257835365+02:00" level=error msg="Handler for POST /v1.23/images/create returned error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: \"<html><body><h1>408 Request Time-out</h1>\\nYour browser didn't send a complete request in time.\\n</body></html>\\n\\n\""

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Reactions: 21
  • Comments: 74 (2 by maintainers)

Most upvoted comments

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

Setting the DNS server to fixed (8.8.8.8) from automatic in docker settings solved the problem for me. dns

I already hat MTU size of 1500 and still not working.

Apparently, enabling MTU probing in the kernel resolved the issue. To enable it just run:

 sudo sysctl -w net.ipv4.tcp_mtu_probing=1

To make this permanently, add the following line to /etc/sysctl.conf:

net.ipv4.tcp_mtu_probing=1

https://thesimplecomputer.info/pages/adventures-in-linux-tcp-tuning-page2

Exactly the same issue here.

@aasaanhai also worked for me. Simply set the default DNS to 8.8.8.8 on the “vEthernet (DockerNAT)” network adapter.

For Windows 10 Home, Putting 8.8.8.8 as the first dns resolves it.

I faced the same problem on a Windows 10 machine. To solve the problem I accessed the Docker client Settings screen, tab Network, and changed the DNS Server to fixed IP 8.8.8.8. After restarting docker everything start working properly.

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 192.168.0.1

after editing to

nameserver 8.8.8.8 nameserver 8.8.4.4 nameserver 10.0.0.10 This worked for me.

[ubuntu16.04, docker 18.01.0] In my case, I could solve by setting proxy like this. This way is different one from above.

$ sudo mkdir -p /etc/systemd/system/docker.service.d $ sudo vi /etc/systemd/system/docker.service.d/http_proxy.conf [Service] Environment=“HTTP_PROXY=http://<your_proxy_ip>:<port>/” $ sudo vi /etc/systemd/system/docker.service.d/https_proxy.conf [Service] Environment=“HTTPS_PROXY=http://<your_proxy_ip>:<port>/” $ sudo systemctl daemon-reload $ sudo systemctl restart docker

I got it solved by adding the proxy details in the docker config file /etc/default/docker.

@luxas I worked for my linux system, but not getting what to do on windows. Please help if possible.

Same problem faced running docker-machine on windows. Restarting docker-machine worked for me suggested by chenrui333. My docker version is 1.13.1

docker-machine restart

docker login as well as docker pull is working now.

I was getting this same error in Windows 10 Home using Docker Toolbox. Changing my DNS to Google’s 8.8.8.8 (secondary to 8.8.4.4) worked for me (per @aasaanhai above). Detailed instructions for how to do so here (see section II).

I’m also getting this issue:

Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Not continuing with pull after error: Network timed out while trying to connect to https://index.docker.io/v1/repositories/armhf/busybox/images. You may want to check your internet connection or if you are behind a proxy.

docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay null bridge host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.19-1-ARCH
Operating System: Arch Linux ARM
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 922.2 MiB
Name: pi3arch
ID: 7MTA:UZIW:SVAS:H3U7:YBHE:FWXV:KWOS:4QAQ:HDLG:EWIV:DOAP:TWFR
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

I’ve set the mtu to 1500 and net.ipv4.tcp_mtu_probing=1 with no success 😦 Any ideas?

Same issue here as reported by @luxas.

I found out that the problem might be in /etc/resolv.conf. I had:

nameserver 10.0.0.10
nameserver 8.8.8.8
nameserver 8.8.4.4

but moving the non-working (yet) 10.0.0.10 nameserver to the bottom fixed it for me I think

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 10.0.0.10

This worked for me thanks

thanks @JohnVonNeumann i solved my issue. the message “Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)” was caused by network connection with a proxy. with https://docs.docker.com/engine/admin/systemd/ (§ http proxy), it is ok. i am surprised docker use systemd on debian… and not environment variables. thanks again

Thank you luxas It works great!!!

I added below configuration on /etc/resolv.conf as suggested above and it works

nameserver 8.8.8.8 nameserver 8.8.4.4 nameserver 10.0.0.10 ​

Please refer to this: https://docs.docker.com/engine/admin/systemd/#httphttps-proxy I have resolved on Ubuntu 16.04 LTS

I’m getting this on Mac OS 10.12.5. I tried re-installing to the latest version of docker, and still no luck. Here’s the error I’m getting: Pulling wordpress (wordpress:latest)… ERROR: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I already hat MTU size of 1500 and still not working.

Apparently, enabling MTU probing in the kernel resolved the issue. To enable it just run:

 sudo sysctl -w net.ipv4.tcp_mtu_probing=1

To make this permanently, add the following line to /etc/sysctl.conf:

net.ipv4.tcp_mtu_probing=1

https://thesimplecomputer.info/pages/adventures-in-linux-tcp-tuning-page2

Thank you… It fixed the issue 💯

docker login to private registry in our environment is failing intermittently. We are having same issue with private registry. Configuring Google’s DNS would work for docker.io but for internal registries how can we extend the login timeout value, esp with slow AD Authorizations.

We have configured our docker NGINX with proxy_ignore_client_abort on; to ignore client timeouts to verify if there is a server kick outs. But the client looks hard coded with 15s timeout irrespective of the server behaviour. Any way we can change/extend this timeout?

I added below configuration on /etc/resolv.conf as suggested above and it works

nameserver 8.8.8.8 nameserver 8.8.4.4 nameserver 10.0.0.10 ​

this one worked for me! thank you so much 😃

Thanks AllDev and Baul,

Let me try this, I will update you.

Thanks Buddy

On Mon, Jul 2, 2018 at 4:32 PM, baul notifications@github.com wrote:

@alldev0825 https://github.com/alldev0825 thanks. For old docker 1.13, can use /etc/sysconfig/docker to set proxy ,but for 17.06 docker-ce, it does not work,need setup proxy in the systemd dir as the link you supplied.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/moby/moby/issues/22635#issuecomment-401768028, or mute the thread https://github.com/notifications/unsubscribe-auth/AGz0zEiSo7eyIYO7opx2bMhAGv1ULj5Fks5uCf3UgaJpZM4IbXBr .

8.8.8.8 didn’t work for me, but 8.8.4.4 did work. looks like whoever got this error- try different dns’s and hopefully one of them will work.

The solution from @brunolm worked for me! Thanks!

As kirstin said setting the DNS on Docker fixes the issue

DNS to 8.8.8.8 on the “vEthernet (DockerNAT)” network adapter.

You can do this from Powershell with

Get-DnsClientServerAddress | Where-Object { $_.InterfaceAlias -like "*(DockerNAT)" } | ForEach-Object { Set-DnsClientServerAddress -InterfaceIndex $_.InterfaceIndex -ServerAddresses @("8.8.8.8", "8.8.4.4") }

@alldev0825, thanks! It works for me.