distribution: S3 certificate must be added to docker daemon registry certificates
I’m trying to set up a Registry v2 backed by AWS S3 that uses self-signed client certificates and no further authentication. My setup so far works with the following (meaning that I can push and pull from it with a client):
- registry running on AWS EC2
- self-signed CA, server and client key
- backed by unsecure S3 (i.e. secure: false in the config)
My config.yml is:
version: 0.1
storage:
cache:
layerinfo: inmemory
s3:
accesskey: REDACTED
secretkey: REDACTED
region: eu-west-1
bucket: REDACTED
encrypt: false
secure: false
v4auth: true
rootdirectory: /registry2
http:
addr: :5000
secret: REDACTED
debug:
addr: :5001
tls:
certificate: /go/src/github.com/docker/distribution/certs/domain.crt
key: /go/src/github.com/docker/distribution/certs/domain.key
clientcas:
- /go/src/github.com/docker/distribution/certs/ca.crt
However, I want the S3 connection to be secured. If I change secure: false to secure: true, it seems that the upload continues as expected, but after a while I get this message (note, this arrives at the client):
# docker push ec2-REDACTED.eu-west-1.compute.amazonaws.com:5000/hello-world:latest
The push refers to a repository [ec2-REDACTED.eu-west-1.compute.amazonaws.com:5000/hello-world] (len: 1)
91c95931e552: Image push failed
FATA[0000] Error pushing to registry: Head https://s3-eu-west-1.amazonaws.com/REDACTED/registry2/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJCQ2SRKBWG3CKEGA%2F20150423%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20150423T185546Z&X-Amz-Expires=1200&X-Amz-Signature=6423a34b004f84d4d0c7dc27b159130a192dc2efcedd43a1bdab081134e60fdd&X-Amz-SignedHeaders=host: x509: certificate signed by unknown authority
On the same machine running docker, I can do:
# curl -v https://s3-eu-west-1.amazonaws.com
* Rebuilt URL to: https://s3-eu-west-1.amazonaws.com/
* Hostname was NOT found in DNS cache
* Trying 54.231.136.64...
* Connected to s3-eu-west-1.amazonaws.com (54.231.136.64) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-SHA
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3-eu-west-1.amazonaws.com
* start date: 2014-10-02 00:00:00 GMT
* expire date: 2015-09-05 23:59:59 GMT
* subjectAltName: s3-eu-west-1.amazonaws.com matched
* issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
* SSL certificate verify ok.
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: s3-eu-west-1.amazonaws.com
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< x-amz-id-2: Zq93t62mf990ntJA69FZXAiSxFl/wtnXq+j63Zaf5DRljHjYH6jz+WKjH+VMfriIeRFmqqt46zI=
< x-amz-request-id: CC1FD79D39F8D877
< Date: Thu, 23 Apr 2015 18:21:06 GMT
< Location: http://aws.amazon.com/s3/
< Content-Length: 0
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
<
* Connection #0 to host s3-eu-west-1.amazonaws.com left intact
The last couple of lines in my registry log is:
INFO[1885] (*layerInfoCache).Fetch("sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4") blob.fetch.duration=5.331µs http.request.host=ec2-REDACTED.eu-west-1.compute.amazonaws.com:5000 http.request.id=b2ea8940-936e-4439-abf8-459a99cbdfc9 http.request.method=HEAD http.request.remoteaddr=10.121.25.208:60849 http.request.uri=/v2/hello-world/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 http.request.useragent=docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-49-generic os/linux arch/amd64 instance.id=3d11069c-6573-46f3-bf4f-b08677af2a36 vars.digest=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 vars.name=hello-world version=v2.0.0-32-g73960f4.m
INFO[1885] Base.URLFor("/docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/data") trace.duration=278.387µs trace.file=/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).URLFor trace.id=4ac3fe2f-16d3-483e-9f76-d40c4d660885 trace.line=161
INFO[1885] response completed http.request.host=ec2-REDACTED.eu-west-1.compute.amazonaws.com:5000 http.request.id=b2ea8940-936e-4439-abf8-459a99cbdfc9 http.request.method=HEAD http.request.remoteaddr=10.121.25.208:60849 http.request.uri=/v2/hello-world/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 http.request.useragent=docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-49-generic os/linux arch/amd64 http.response.duration=2.846826ms http.response.status=307 http.response.written=0 instance.id=3d11069c-6573-46f3-bf4f-b08677af2a36 version=v2.0.0-32-g73960f4.m
10.121.25.208 - - [23/Apr/2015:18:55:46 +0000] "HEAD /v2/hello-world/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 307 0 "" "docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-49-generic os/linux arch/amd64"
I really have no idea what’s going on here. Is it possible that somehow the S3 storage driver is using my certificate or CA chain to try to verify the https response from S3 (and the error is forwarded to my client)? Or is this a problem in the docker client (but why will that end up knowing about S3)?
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 24 (5 by maintainers)
Ok, so your comment convinced me to experiment a bit further on the client daemon side. I now have it running. The missing piece was that /etc/docker/certs.d/REGISTRY/ca.crt MUST contain the cert for s3 as well (it seems that the docker daemon uses the ca.crt to check validity of the https from s3 instead of using the ca-certificates.crt file). This does not seem to be documented. So far, I just had the ca.crt contain our self-generated ca.crt.
TL;DR for others: The fix is to append your own ca.crt to a complete ca-certificates.crt and use the result as /etc/docker/certs.d/REGISTRY/ca.crt.
On Ubuntu 14.04, doing the following on each server got it working for me:
Where docker-registry:5000 is the name of my docker registry server.
Just make sure your private certificate is in the same directory with the new link. It looks like overriding the client CAs causes docker client to no longer look at the system installed CAs anymore.
vim common/templates/registry/config.yml storage: s3: accesskey: xxxxxxxxxx secretkey: xxxxxxxxxx regionendpoint: http://xxx.xxx.xxx.xxx bucket: docker-registry region: Regionone #rootdirectory: /s3/object/name/prefix #encrypt: true #secure: true #v4auth: true #chunksize: 5242880 #multipartcopychunksize: 33554432 #multipartcopymaxconcurrency: 100 #multipartcopythresholdsize: 33554432
docker-compose stop ; docker-compose up -d
docker tag centos hub.xxxx.com.cn/sys-admin/centos
docker push hub.xxxx.com.cn/sys-admin/centos 5.Errror 👍 172.10.10.51 - - [28/Jun/2017:16:53:21 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/data?partNumber=3&uploadId=2~MIIapfmrwZ7DLkAxeBiPj0bdc5bS-0Q HTTP/1.1” 200 25 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:24 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/data?partNumber=4&uploadId=2~MIIapfmrwZ7DLkAxeBiPj0bdc5bS-0Q HTTP/1.1” 200 25 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:26 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/data?partNumber=5&uploadId=2~MIIapfmrwZ7DLkAxeBiPj0bdc5bS-0Q HTTP/1.1” 200 25 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/hashstates/sha256/72267384 HTTP/1.1” 200 0 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/data?partNumber=6&uploadId=2~MIIapfmrwZ7DLkAxeBiPj0bdc5bS-0Q HTTP/1.1” 200 25 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/hashstates/sha256/72267384 HTTP/1.1” 200 0 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/startedat HTTP/1.1” 206 20 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry?prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fsys-admin%2Fcentos%2F_uploads%2Fa550c481-457b-4d1a-bea9-a2acb55152f5%2Fdata&uploads= HTTP/1.1” 200 1033 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/data?uploadId=2~MIIapfmrwZ7DLkAxeBiPj0bdc5bS-0Q HTTP/1.1” 200 1596 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “POST http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/data?uploadId=2~MIIapfmrwZ7DLkAxeBiPj0bdc5bS-0Q HTTP/1.1” 200 350 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/hashstates/sha256/0 HTTP/1.1” 200 0 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry?max-keys=1&prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fsys-admin%2Fcentos%2F_uploads%2Fa550c481-457b-4d1a-bea9-a2acb55152f5%2Fdata HTTP/1.1” 200 718 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry?delimiter=%2F&max-keys=1000&prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fsys-admin%2Fcentos%2F_uploads%2Fa550c481-457b-4d1a-bea9-a2acb55152f5%2Fhashstates%2Fsha256%2F HTTP/1.1” 200 1152 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry/docker/registry/v2/repositories/sys-admin/centos/_uploads/a550c481-457b-4d1a-bea9-a2acb55152f5/hashstates/sha256/72267384 HTTP/1.1” 206 201 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry?max-keys=1&prefix=docker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2Fd3%2Fd3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0%2Fdata HTTP/1.1” 200 353 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry?max-keys=1&prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fsys-admin%2Fcentos%2F_uploads%2Fa550c481-457b-4d1a-bea9-a2acb55152f5%2Fdata HTTP/1.1” 200 718 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “GET http://172.10.10.20/docker-registry?max-keys=1&prefix=docker%2Fregistry%2Fv2%2Frepositories%2Fsys-admin%2Fcentos%2F_uploads%2Fa550c481-457b-4d1a-bea9-a2acb55152f5%2Fdata HTTP/1.1” 200 718 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “POST http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?uploads= HTTP/1.1” 200 362 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=3&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=1&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=2&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=3&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=1&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=2&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=3&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=2&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:29 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=1&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)” 172.10.10.51 - - [28/Jun/2017:16:53:30 +0800] “PUT http://172.10.10.20/docker-registry/docker/registry/v2/blobs/sha256/d3/d3aeceeb0289bf97c2474366cfcdaf66e29b409d45d6ed91be3335f16915f1b0/data?partNumber=2&uploadId=2~jgznOIk_bokLV8qnElK0FIAgHetKXPc HTTP/1.1” 501 231 “-” “aws-sdk-go/1.2.4 (go1.7.5; linux; amd64)”
check ceph s3 service : [root@XXXXX ~]# s3cmd put salt-2014.7.5-1.el7.noarch.rpm s3://docker-registry/s3/object/name/prefix/docker/registry/v2/repositories/sys-admin/ upload: ‘salt-2014.7.5-1.el7.noarch.rpm’ -> ‘s3://docker-registry/s3/object/name/prefix/docker/registry/v2/repositories/sys-admin/salt-2014.7.5-1.el7.noarch.rpm’ [1 of 1] 3396656 of 3396656 100% in 0s 35.25 MB/s done
@dmcgowan I agree completely. As a user, I would expect the ca.crt to be used in addition to the system certificates. As an aside, I tried putting my own ca.crt as a system certificate, but had no luck with that approach.
@endophage I like your whitelist idea, but feel that that should probably be a separate feature (I mean e.g. putting file whitelist_only in
/etc/docker/certs.d/REGISTRY). It does not feel like that should be the default behaviour, or alternatively it should be documented precisely 😃.