moby: Docker version 1.8.2 does not send auth header with PATCH HTTP request when pushing to default port

This issue is related to https://github.com/docker/docker/issues/16728 , but bear with me please. (bug report at the bottom)

Environment: Docker registry (Artifactory) behind NginX. Port 443 (standard HTTPS) redirects to the Docker endpoint and should be used to access it. I am logged in through docker login

The problem:

• docker push docker.int.domain.com:443/some-image works

• docker push docker.int.domain.com/some-image does not. The output:

    The push refers to a repository [docker.int.domain.com/testtest/ubuntu] (len: 1)
    8d885ef1e201: Pushing [==================================================>]  30.9 MB
    unauthorized: The client does not have permission to push to the repository.
  

I had a look at NginX’s access logs and this is what I found:

• When pushing to the “port-less” address, the request is directed to the port 443, so those commands should behave the same way.
• The PATCH HTTP request does not contain auth header, yet all other requests do. This causes the push command to fail with unauthorized: The client does not have permission to push to the repository.

This issue is present in version 1.8.2. With version 1.7.x pushing without explicitly specifying the port was possible.

NginX’s access log:

• The second column is the name of the authenticated user
• p: denotes port
• The last number is response HTTP code

docker push docker.int.domain.com/some-image

192.168.33.3 , - , [07/Dec/2015:14:32:04 +0100] , p:443 , "GET /v2/", 401
192.168.33.3 , melka , [07/Dec/2015:14:32:06 +0100] , p:443 , "HEAD /v2/testtest/ubuntu/(...)", 404
192.168.33.3 , melka , [07/Dec/2015:14:32:07 +0100] , p:443 , "POST /v2/testtest/ubuntu/(...)", 202
192.168.33.3 , - , [07/Dec/2015:14:33:08 +0100] , p:443 , "PATCH /v2/testtest/ubuntu/(...)" , 403

docker push docker.int.domain.com:443/some-image

192.168.33.3 , - , [07/Dec/2015:14:43:23 +0100] , p:443 , "GET /v2/", 401
192.168.33.3 , melka , [07/Dec/2015:14:43:24 +0100] , p:443 , "HEAD /v2/testtest/ubuntu/(...)", 404
192.168.33.3 , melka , [07/Dec/2015:14:43:24 +0100] , p:443 , "POST /v2/testtest/ubuntu/(...) HTTP/1.1", 202
192.168.33.3 , melka , [07/Dec/2015:14:43:27 +0100] , p:443 , "PATCH /v2/testtest/ubuntu/(...)", 202
192.168.33.3 , melka , [07/Dec/2015:14:43:27 +0100] , p:443 , "PUT /v2/testtest/ubuntu/(...)", 201
(...)

As you can see, for some reason when pushing without specifying port, the name is empty. That causes the push to fail. When a port is set, the push continues with more log messages below.

Just to be on the safe side, here is my NginX config: (I have tried removing the X-Forwarded-headers as suggested here https://docs.docker.com/registry/nginx/ - that did not help).

    server {
        listen 80;
        server_name docker.int.domain.com;
        return 301 https://$server_name$request_uri;
    }

    server {
        listen 443;
        server_name docker.int.domain.com;

        ssl on;
        ssl_certificate /etc/nginx/cert/artifactory-ca.crt;
        ssl_certificate_key /etc/nginx/cert/artifactory-ca.key;

        access_log  /var/log/nginx/docker.int.domain.com.443.access.log  myformat;
        error_log  /var/log/nginx/docker.int.domain.com.443.error.log;

        proxy_set_header Host $host:443;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Original-URI $request_uri;
        proxy_pass_header Server;  # To help debugging, list the server that actually did the reply rather than nginx
        proxy_read_timeout 900;

        client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads

        # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
        chunked_transfer_encoding on;

        location /v2 {
            proxy_pass http://localhost:8081/artifactory/api/docker/docker/v2;
        }
    }

We want to be able to refer to the repository just by the server name, without having to specify the port. But right now, that is only possible using the older docker client.

Thanks for taking a look at this

BUG REPORT INFORMATION

docker version:

Client:
 Version:      1.8.2
 API version:  1.20
 Package Version: docker-1.8.2-7.el7.centos.x86_64
 Go version:   go1.4.2
 Git commit:   bb472f0/1.8.2
 Built:        
 OS/Arch:      linux/amd64

docker info:

Containers: 3
Images: 17
Storage Driver: devicemapper
 Pool Name: docker-253:0-69269427-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 4.353 GB
 Data Space Total: 107.4 GB
 Data Space Available: 8.753 GB
 Metadata Space Used: 3.731 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.10.0-229.14.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 3.703 GiB
Name: localhost.localdomain
ID: UGHD:SKNU:RRGU:GERL:KGDA:3Q7I:T5L5:LWGO:AM2Z:LTRX:MIX4:XBK3
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

uname -a:

Linux localhost.localdomain 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Running in VirtualBox.

List the steps to reproduce the issue:

1. Have a registry behind NginX (I have Artifactory-based)
2. Set up NginX so that the registry is accessible on port 443.
3. Push an image to the registry, without specifying the port number

Describe the results you received:

• I expect the image to be pushed successfully, because port 443 is the default.

Describe the results you expected:

• The push fails with an error: unauthorized: The client does not have permission to push to the repository.

----------END REPORT ---------