distribution: docker push keeps retrying and failing to local registry

I’m also hitting this issue. Unless someone can point to the fix, I would still consider this an open issue.

This gave me the answer that worked for me – in short, ensure that you run your Registry container with the REGISTRY_HTTP_HOST env var specified:

docker run -e REGISTRY_HTTP_HOST=https://my.registry:5000 -d -p 5000:5000 --name registry registry:2

I’m also seeing this. docker service logs also shows no recognizable errors on the side of the registry, but eventually the push fails with a 500 internal server error.

i think i have a solution , make sure that you have the correct tag name for the image , i think this will fix this issue

I have the same issue.

controller-0:~$ sudo docker push 192.168.31.49:5000/docker.io/nfvpe/multus:v3.4 
The push refers to repository [192.168.31.49:5000/docker.io/nfvpe/multus] 
b39e09a96a41: Retrying in 14 seconds 
b4b5c7759f04: Retrying in 13 seconds 
877b494a9f30: Retrying in 13 seconds 

Please check your docker proxy settings in /etc/systemd/system/docker.service.d/http-proxy.conf. If you set the proxy, you need set no proxy entry.

[Service]
Environment="HTTP_PROXY=http://[your proxy]:[port]/" "HTTPS_PROXY=http://[your proxy]:[port]/" "NO_PROXY=localhost,127.0.0.1,[your private docker registry]"

Another way, disable the proxy file with mv http-proxy.conf http-proxy.bak directly .

Notice，after you changed your proxy file. you have to reload the docker service. sudo systemctl restart docker sudo systemctl daemon-reload

This gave me the answer that worked for me – in short, ensure that you run your Registry container with the REGISTRY_HTTP_HOST env var specified:

docker run -e REGISTRY_HTTP_HOST=https://my.registry:5000 -d -p 5000:5000 --name registry registry:2

@bachya Thanks so much for this! For some reason I can’t seem to find any real docs on REGISTRY_HTTP_HOST anywhere. Do you recall where you found the documentation for this?

It happened to me, the issue was that i had no permits to create a new image, if you wanna overcome without adding permissions, upload using a different label to your image, and you should be fine =) if nt, just add permissions to your role =)

It was missing IAM permission on the user in my case.

I had a very similar error (retrying until failure) and it was not related to docker, but to the file system of the machine hosting the registry. In my case the logs were almost identical to yours except that they said

msg="mkdir: [...] permission denied"

I solved it by running the update-permissions gitlab script.

Now your case seems to be a little different. From your logs:

msg="unknown error reading request payload: unexpected EOF"

I’d take a look at the registry’s directory (in my case it was /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/) and inspect what’s going on there. Have a look at the permissions and search for corrupt files. I hope this helps.

I had similar issue where pushing kept getting stuck at retrying to push layers to registry. Problem was fixed by me noticing the server’s disk space was full and making space for it. Docker is not informing this in anyway here.

It kept trying that same layer every 10 seconds.

PS H:\> docker push <server_ip>:<port>/<image_name>
The push refers to repository [<server_ip>:<port>/<image_name>]
3da6320debf3: Preparing
4039e516f7dc: Preparing
a630bea6a912: Preparing
d72d108ae894: Preparing
2f650ec61ae2: Preparing
3da6320debf3: Retrying in 8 seconds
06ccd0d4c4fa: Layer already exists
bbd1b85c7918: Layer already exists
d121aacf7b59: Layer already exists
0f905224ac84: Layer already exists
e492023cc4f9: Layer already exists
cbda574aa37a: Layer already exists
8451f9fe0016: Layer already exists
858cd8541f7e: Layer already exists
a42d312a03bb: Layer already exists
dd1eb1fd7e08: Layer already exists