distribution: Private registry push fails (S3): Error with Blob unknown to registry

Hi,

We are running with the latest Registry 2.4.0 and trying to use S3 as the backend. When pushing an image to the registry, it keeps retrying until ultimately ending in “Image push failed”. Once the image push has failed the following errors are available:

docker logs registry | grep -E '500 |error'

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:c807ad6f343636e38ac12b5cf4cc90529a8e26e295f9f5f5a746a269b64f9d74 err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=f2ba32b9-c0d8-43ae-aa76-4f8061bc4cc8 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56405" http.request.uri="/v2/webgateway/blobs/sha256:c807ad6f343636e38ac12b5cf4cc90529a8e26e295f9f5f5a746a269b64f9d74" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.143871ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:c807ad6f343636e38ac12b5cf4cc90529a8e26e295f9f5f5a746a269b64f9d74" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:7a90e5079b15df521550cfa6ba5f2ae07a60f08fcb081560bafc6418a22c9faa err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=0d0c45d2-89d2-4580-8215-34fba6a0266a http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56407" http.request.uri="/v2/webgateway/blobs/sha256:7a90e5079b15df521550cfa6ba5f2ae07a60f08fcb081560bafc6418a22c9faa" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=14.924243ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:7a90e5079b15df521550cfa6ba5f2ae07a60f08fcb081560bafc6418a22c9faa" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:bfe199ea6ecf24ff769f2c9c0a65eba5f2816e6a11c7c8c2b36b7a6da0087e0d err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=6e4bad09-ce07-4811-bac6-176ed6192a54 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56409" http.request.uri="/v2/webgateway/blobs/sha256:bfe199ea6ecf24ff769f2c9c0a65eba5f2816e6a11c7c8c2b36b7a6da0087e0d" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=31.48559ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:bfe199ea6ecf24ff769f2c9c0a65eba5f2816e6a11c7c8c2b36b7a6da0087e0d" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:9f8ccdfa573d9fe166cb9f20e1d05eec768f648879c519eabe686f1f75ab6cf6 err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=e3c04a84-ed10-433d-a579-95b6a4399b75 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56412" http.request.uri="/v2/webgateway/blobs/sha256:9f8ccdfa573d9fe166cb9f20e1d05eec768f648879c519eabe686f1f75ab6cf6" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=24.663973ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:9f8ccdfa573d9fe166cb9f20e1d05eec768f648879c519eabe686f1f75ab6cf6" vars.name=webgateway version=v2.4.0

time="2016-04-15T06:11:36Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:683680549799915daca7cd05676b1d96ad05e63423d074d5aefd7d9240dc9a83 err.message="blob unknown to registry" go.version=go1.6.1 http.request.host="docker.dev.ourown.net:5000" http.request.id=8c6e5366-a2d1-4294-bd03-67c98efb4674 http.request.method=HEAD http.request.remoteaddr="10.40.110.63:56416" http.request.uri="/v2/webgateway/blobs/sha256:683680549799915daca7cd05676b1d96ad05e63423d074d5aefd7d9240dc9a83" http.request.useragent="docker/1.11.0 go/go1.5.4 git-commit/4dc5990 kernel/3.13.0-48-generic os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.0 \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=32.042124ms http.response.status=404 http.response.written=157 instance.id=cf4d4f88-151e-4c73-82f5-f378294f2d76 vars.digest="sha256:683680549799915daca7cd05676b1d96ad05e63423d074d5aefd7d9240dc9a83" vars.name=webgateway version=v2.4.0

I can see it creates buckets in S3, so it has connection to it - the configuration is as follows:

docker run \
         -v `pwd`/certs:/certs \
         -e "REGISTRY_STORAGE=s3" \
         -e "REGISTRY_STORAGE_S3_REGION=eu-west-1" \
         -e "REGISTRY_STORAGE_S3_BUCKET=docker-registry" \
         -e "REGISTRY_STORAGE_S3_ACCESSKEY=[OWNKEY]" \
         -e "REGISTRY_STORAGE_S3_SECRETKEY=[OWNKEY]" \
         -e "REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt" \
         -e "REGISTRY_HTTP_TLS_KEY=/certs/domain.key" \
         -d \
         -p 5000:5000 \
         registry:2.4.0