moby: Failed to push one large image, "unauthorized: authentication required"

I am trying to push several images to docker hub referring to this document: Push images to Docker Cloud and I succeed in pushing all images(quite small, less than 100 MB) except one large image(about 5G). The terminal reports the error message: “unauthorized: authentication required”.

Any suggestion will be appreciated!

Steps to reproduce the issue:

  1. export DOCKER_ID_USER=“username”
  2. docker login
  3. docker tag my_image $DOCKER_ID_USER/my_image
  4. docker push $DOCKER_ID_USER/my_image

Describe the results you received: The terminal reports the error message: “unauthorized: authentication required”.

I suspect that the login gets invalid when pushing the large image. However, after this failure, I tried to push a small image and it works, which indicates that the login is still valid.

I also suspect that the large size of the image is the root cause but all info I get suggests that Docker Hub has no size limitaion.

Additional information you deem important (e.g. issue happens only occasionally):*

Output of docker version:

Docker version 1.12.6, build 3a094bd/1.12.6

Output of docker info:

Containers: 7
 Running: 1
 Paused: 0
 Stopped: 6
Images: 519
Server Version: 1.12.6
Storage Driver: devicemapper
 Pool Name: docker-253:0-2550137067-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 53.69 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 67.11 GB
 Data Space Total: 107.4 GB
 Data Space Available: 40.26 GB
 Metadata Space Used: 61.31 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.086 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 8
Total Memory: 31.14 GiB
Name: localhost.localdomain
ID: 6C7Z:Y2ML:4MZN:QS3S:FVE7:5GNW:SYMU:RZZX:CLBD:QUUL:VIRR:5OYN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Http Proxy:***
Https Proxy: ***
Username:******
Registry: https://index.docker.io/v1/
Insecure Registries:
 **.***.**.***:15000
 **.***.**.***:15000
 ***.*.*.*/8
Registries: docker.io (secure)

Docker daemon log

-- Logs begin at Mon 2017-04-24 23:25:05 EDT. --
Jun 06 16:45:53 localhost.localdomain dockerd-current[28490]: time="2017-06-06T16:45:53.187898540-04:00" level=info msg="{Action=push, Usern  ame=arda, LoginUID=1000, PID=9294}"
Jun 06 16:46:44 localhost.localdomain dockerd-current[28490]: time="2017-06-06T16:46:44.004882036-04:00" level=info msg="{Action=auth, Usern  ame=arda, LoginUID=1000, PID=9328}"
Jun 06 16:47:23 localhost.localdomain dockerd-current[28490]: time="2017-06-06T16:47:23.886609210-04:00" level=info msg="{Action=tag, Userna  me=arda, LoginUID=1000, PID=9359}"
Jun 06 16:47:32 localhost.localdomain dockerd-current[28490]: time="2017-06-06T16:47:32.428378688-04:00" level=info msg="{Action=push, Usern  ame=arda, LoginUID=1000, PID=9365}"
Jun 06 16:59:48 localhost.localdomain dockerd-current[28490]: time="2017-06-06T16:59:48.150693999-04:00" level=error msg="Upload failed: una  uthorized: authentication required"
Jun 06 16:59:48 localhost.localdomain dockerd-current[28490]: time="2017-06-06T16:59:48.151824766-04:00" level=error msg="Attempting next en  dpoint for push after error: unauthorized: authentication required"
Jun 06 17:11:47 localhost.localdomain dockerd-current[28490]: time="2017-06-06T17:11:47.292123061-04:00" level=info msg="{Action=push, Usern  ame=arda, LoginUID=1000, PID=9774}"
Jun 06 18:18:31 localhost.localdomain dockerd-current[28490]: time="2017-06-06T18:18:31.319974509-04:00" level=info msg="{Action=auth, Usern  ame=arda, LoginUID=1000, PID=10681}"
Jun 06 18:19:44 localhost.localdomain dockerd-current[28490]: time="2017-06-06T18:19:44.807555097-04:00" level=info msg="{Action=tag, Userna  me=arda, LoginUID=1000, PID=10698}"
Jun 06 18:19:50 localhost.localdomain dockerd-current[28490]: time="2017-06-06T18:19:50.833667618-04:00" level=info msg="{Action=push, Usern  ame=arda, LoginUID=1000, PID=10704}"

About this issue

  • Original URL
  • State: open
  • Created 7 years ago
  • Reactions: 2
  • Comments: 28 (6 by maintainers)

Most upvoted comments

This issue appears to be noted here: https://gitlab.com/gitlab-org/gitlab-ce/issues/19867

And states that an authentication token time can be set on the docker registry to be valid for a longer time. It defaults to 5 minutes.

I think I am being affected as well. I do docker login, but there is one large layer (several GB) that takes a few minutes to upload, and as soon as it’s done uploading I get “unauthorized: authentication required” even though pushing smaller images works fine

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:58 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:24 2018
  OS/Arch:          linux/amd64
  Experimental:     false

I don’t know why the auth token is so short-lived or at least, why it can’t be renewed. Seems to be a legit bug.

I’ve tried a dozen times now and I don’t know what to do. I guess I’ll have to see if I can hack together a way to break this one image layer up into smaller layers. But there are good reasons it’s large and splitting it up will have to require some hacks.

I am also having this exact same issue (18.03.1-ce linux).

There is some info that I forgot to mention. The large layer 0b339c269448 is the last layer to push, before which, all layers are pushed to the docker hub successfully. And the message “unauthorized: authentication required” appeared as soon as the large layer finished pushing(The arrow reached the end and the size became 3.855G). I have pushed this image several times, and I got the unauthorized message every time when the “====>” arrow reached the right end.