distribution: private registry delete manifest results in broken state

Saw some similar issues, but didnt find anything for this particular problem.

docker registry version: registry:2 Digest: sha256:bf9b4a7b53a2f54c7b4d839103ca5be05b6a770ee0ba9c43e9ef23d602414f44

I am using a private registry backed by S3, using basic auth.

Steps:

list tags do: curl https://<regsitry>/v2/manager/tags/list observe: {"name":"manager","tags":[...,"1.3.0","1.3.1","latest"]}

I want to delete tag 1.3.1

get tag digest do: curl https://<registry>/v2/manager/manifests/1.3.1 -v observe: < Docker-Content-Digest: sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a
delete tag do: curl -X DELETE https://<registry>/v2/manager/manifests/sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a -v observe: < HTTP/1.1 202 Accepted
ensure tag is gone do: curl https://<regsitry>/v2/manager/tags/list observe: {"name":"manager","tags":[...,"1.3.0","1.3.1","latest"]} server logs:

INFO[0907] response completed                            go.version=go1.6.2 http.request.host=<registry> http.request.id=63c217c1-b5ee-4d67-9a19-8fd79f210d87 http.request.method=GET http.request.remoteaddr=<my ip>:53486 http.request.uri=/v2/manager/tags/list http.request.useragent=curl/7.43.0 http.response.contenttype=application/json; charset=utf-8 http.response.duration=52.464518ms http.response.status=200 http.response.written=200 instance.id=6484f785-0dfd-4300-afde-13e8ad9da085 version=v2.4.1
67.161.4.204 - - [31/May/2016:00:24:31 +0000] "GET /v2/manager/tags/list HTTP/1.1" 200 200 "" "curl/7.43.0"

investigate why tag is still there do: question life and the meaning of happiness, curl https://<registry>/v2/manager/manifests/1.3.1 -v observe: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"manager","Revision":"sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a"}}]} server logs:

ERRO[0940] response completed with error                 auth.user.name=<username> err.code=manifest unknown err.detail=unknown manifest name=manager revision=sha256:4cd37f5633ac95dc32ceca3359a109f2c5a20517f2ded9eace4e12dccf58835a err.message=manifest unknown go.version=go1.6.2 http.request.host=<registry> http.request.id=2a33440c-2d08-49d0-9ee0-61a9e6cf4996 http.request.method=GET http.request.remoteaddr=<my ip>:53492 http.request.uri=/v2/manager/manifests/1.3.1 http.request.useragent=curl/7.43.0 http.response.contenttype=application/json; charset=utf-8 http.response.duration=30.85648ms http.response.status=404 http.response.written=183 instance.id=6484f785-0dfd-4300-afde-13e8ad9da085 vars.name=manager vars.reference=1.3.1 version=v2.4.1
67.161.4.204 - - [31/May/2016:00:25:04 +0000] "GET /v2/manager/manifests/1.3.1 HTTP/1.1" 404 183 "" "curl/7.43.0"

Questions: How do I actually remove this tag?

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 31 (7 by maintainers)

Most upvoted comments

FYI This command

 curl -v https://<registry>/v2/<repo>/manifests/<tag>

does not provide the right tag digest for v2 deletion. You have to use a additional header as mentionned in the offical docs: https://docs.docker.com/registry/spec/api/#/deleting-an-image

curl -v https://<registry>/v2/<repo>/manifests/<tag> -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'

Proceeding this way fixed the issue for me.

+21

sen-lgtm on Jul 13, 2016

We’re experiencing this bug in Registry v2.6.1.

curl -X GET "localhost:5000/v2/thirdparty/gitlab/tags/list"
{"name":"thirdparty/gitlab","tags":["8.17","9.0.5","9.1.1","9.1.2"]}

returns 4 tags, when fetching manifests of tag 8.17 as https://github.com/docker/distribution/issues/1755#issuecomment-232329426 said, it failed:

curl -v "localhost:5000/v2/thirdparty/gitlab/manifests/8.17" -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> GET /v2/thirdparty/gitlab/manifests/8.17 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: application/vnd.docker.distribution.manifest.v2+json
>
< HTTP/1.1 404 Not Found
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Date: Sat, 06 May 2017 06:49:36 GMT
< Content-Length: 193
<
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"thirdparty/gitlab","Revision":"sha256:1f2291aa0094f379a331772b3e34dfd68d26594055388d74877d5d05b2de72c0"}}]}
* Connection #0 to host localhost left intact

docker logs of registry as below:

time="2017-05-06T06:54:53Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=thirdparty/gitlab revision=sha256:1f2291aa0094f379a331772b3e34dfd68d26594055388d74877d5d05b2de72c0" err.message="manifest unknown" go.version=go1.7.3 http.request.host="localhost:5000" http.request.id=dff260f1-b89a-450a-bad5-7446fd7f7283 http.request.method=GET http.request.remoteaddr="172.18.0.1:50522" http.request.uri="/v2/thirdparty/gitlab/manifests/8.17" http.request.useragent="curl/7.35.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=45.967892ms http.response.status=404 http.response.written=193 instance.id=c02efd08-a6e7-4ac0-86ce-75427ca2d6bb vars.name="thirdparty/gitlab" vars.reference=8.17 version=v2.6.1
172.18.0.1 - - [06/May/2017:06:54:53 +0000] "GET /v2/thirdparty/gitlab/manifests/8.17 HTTP/1.1" 404 193 "" "curl/7.35.0"

How can I delete this tag now?

jostyee on May 6, 2017

Hi all, thanks for reporting the issue. Unfortunately I was unable to reproduce the issue.

I made a minimal config.yml where I deployed registry with: docker run --rm -it -v $(pwd)/config.yml:/etc/docker/registry/config.yml -p 5000:5000 --name registry registry:2.6.2

Configuration:

version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  s3:
    region: <stripped>
    bucket: <stripped>
    accesskey: <stripped>
    secretkey: <stripped>
http:
  addr: :5000

My steps:

$ docker pull alpine:3.5
$ docker pull alpine:3.6
$ docker tag alpine 3.5 localhost:5000/some/repo:1
$ docker tag alpine 3.6 localhost:5000/some/repo:2
$ docker push localhost:5000/some/repo:1
$ docker push localhost:5000/some/repo:2

$ curl localhost:5000/v2/some/repo/tags/list
{"name":"some/repo","tags":["1","2"]}
$ curl -v "localhost:5000/v2/some/repo/manifests/1" -H 'Accept: application/vnd.docker.distribution.manifest.v2+json'
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 5000 (#0)
> GET /v2/some/repo/manifests/1 HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.54.1
> Accept: application/vnd.docker.distribution.manifest.v2+json
>
< HTTP/1.1 200 OK
< Content-Length: 528
< Content-Type: application/vnd.docker.distribution.manifest.v2+json
< Docker-Content-Digest: sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b
< Docker-Distribution-Api-Version: registry/2.0
< Etag: "sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b"
< Date: Wed, 04 Oct 2017 21:22:20 GMT
<
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1511,
      "digest": "sha256:37c7be7a096b71f4e12e957c5ac83c3a7b01d8ae9f43a68c319e298d61b608fe"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1970271,
         "digest": "sha256:ab14e39f58e6d8ba465d2bb577a82a750ec0bcd2342b380920f9e7f307be3c4f"
      }
   ]
* Connection #0 to host localhost left intact
}

$ curl -X DELETE -v localhost:5000/v2/some/repo/manifests/sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 5000 (#0)
> DELETE /v2/some/repo/manifests/sha256:b12d242b6f881c97dbed5205d130b3b17307595c90b27cee79b1b3ea34cd9e2b HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/7.54.1
> Accept: */*
>
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0
< Date: Wed, 04 Oct 2017 21:23:15 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact 

$ curl localhost:5000/v2/some/repo/tags/list
{"name":"some/repo","tags":["2"]}

If you are still experiencing problems, please open a new issue and provide your configuration and steps to reproduce the issue. It’s likely there are differences between my minimal setup and specific configurations that triggering the bug.

hinshun on Oct 4, 2017

Sounds like this is fixed in 2.5.1. Closing.

If anyone is still having issues, please let me know.

stevvooe on Jan 10, 2017

@sudo-bmitch by the way, regctl succeeded to delete anything properly 👏 : I will look at regbot that sounds great for automation. Thank you.

I also found this cleanup.sh script that works on the local filesystem. It tends to give information on the file structure.

At the end, upgrading from 2.6 to 2.7 allowed to use the new /bin/registry garbage-collect -m option that freed up 75% of the storage!

jcarsique on Mar 2, 2021

@jcarsique There’s also the OCI media types. It’s a bit self promoting, but regclient’s regctl is what I use for this, lot of similar functionality to reg. There, the --list flag is needed when running image digest to get the correct digest to delete.

Double check that the digest you deleted matches what you’re seeing in those repos, e.g.

curl -s -X DELETE http://registry:5000/v2/REPO/manifests/sha256:9ef53bd1a6a6e2c8827250315d52cac6fd9a5623831984caa08e3910e66c66ef

Once the manifest is removed for me, the tag is gone immediately without any GC. The GC is needed to remove some of the underlying blobs. I don’t know enough of the filesystem storage of registry to be able to advise what’s safe to remove.

sudo-bmitch on Mar 1, 2021

@jcarsique the header for a docker manifest list is Accept: application/vnd.docker.distribution.manifest.list.v2+json. Here’s an example cleaning v0.2.0 from this local repo:

$ curl -s http://localhost:5000/v2/regclient/regsync/tags/list
{"name":"regclient/regsync","tags":["v0.2.0","copy","latest"]} 

$ curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
{       
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",       
   "schemaVersion": 2, 
   "config": {    
      "mediaType": "application/vnd.docker.container.image.v1+json",        
      "digest": "sha256:55dff1bf550e642373247f3d30ae809d2ba353dfaba89c85e2496e1cbf1e4550",
      "size": 3025                  
   },                                                               
   "layers": [                                                                       
...

$ curl -s http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0                                                                                                                                                                                                           
{                                                                                                                                                                                                                                                                               
   "schemaVersion": 1,                                                                                                                                                                                                                                                          
   "name": "regclient/regsync",                                                                                                                                                                                                                                                 
   "tag": "v0.2.0",                                                                                                                                                                                                                                                             
   "architecture": "amd64",                                                                                                                                                                                                                                                     
   "fsLayers": [                                                                                                                                                                                                                                                                
...

$ curl -s -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0                                                                                                                                    
{                                                                                                                                                                                                                                                                               
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",                                                                                                                                                                                                    
   "schemaVersion": 2,                                                                                                                                                                                                                                                          
   "manifests": [                                                                                                                                                                                                                                                               
      {                                                                                                                                                                                                                                                                         
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                                                                                                                                                                                                   
         "digest": "sha256:615f1aa6e32c5596668a30b31f123e5920eb15ffff78710722f2c6afa8a8d8ee",                                                                                                                                                                                   
         "size": 1152,                                                                                                                                                                                                                                                          
         "platform": {                                                                                                                                                                                                                                                          
            "architecture": "386",                                                                                                                                                                                                                                              
            "os": "linux"                                                                                                                                                                                                                                                       
         }                                                                                                                                                                                                                                                                      
      },                                                             
      {                                                                
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",                 
         "digest": "sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e",                                                                                                                                                                                   
         "size": 1152,                                                         
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
...

Same image, three different manifests depending on the header, and the manifest list is the real one, the registry is looking up the default platform for the v2 manifest, and automatically converting that to a v1 schema without any header. Those manifests have different digests:

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 2342
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Docker-Content-Digest: sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:34:48 GMT

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 1152
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:34:55 GMT

Deleting the digest for a single platform leaves things in a semi-broken state:

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 1152
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:34:55 GMT

$ curl -s -X DELETE http://localhost:5000/v2/regclient/regsync/manifests/sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 404 Not Found
Content-Type: application/json
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:36:24 GMT
Content-Length: 193

$ curl -s http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"regclient/regsync","Revision":"sha256:2e906ee0ad8a6a8182dc824cd4d231e61fa9f4982f417abc32a17d677eda866e"}}]}

$ curl -s http://localhost:5000/v2/regclient/regsync/tags/list
{"name":"regclient/regsync","tags":["v0.2.0","copy","latest"]}

But you can still get the manifest list digest and remove that to clean things up:

$ curl -s -I -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' http://localhost:5000/v2/regclient/regsync/manifests/v0.2.0
HTTP/1.1 200 OK
Content-Length: 2342
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Docker-Content-Digest: sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b"
X-Content-Type-Options: nosniff
Date: Mon, 01 Mar 2021 11:36:41 GMT

$ curl -s -X DELETE http://localhost:5000/v2/regclient/regsync/manifests/sha256:41ed41d4386375daed975741a71033cacea65973f1b4cb06aac8333a73c6c34b

$ curl -s  http://localhost:5000/v2/regclient/regsync/tags/list
{"name":"regclient/regsync","tags":["copy","latest"]}

sudo-bmitch on Mar 1, 2021

I can confirm that we are also seeing the same issue with our private registry (version 2.7.1) and using the proper headers, etc.

Note that this doesn’t happen on every image deletion. Most deletes are successful and remove the manifest/tag together but some get “stuck” in this state. One workaround solution that we have found is to push another “dummy” image to the “stuck” tag and then retry the deletion. This has worked so far, but I’m unsure if the blobs from the old tag are actually deleted on the storage or have become orphaned by this “workaround”.

a1kemist on Apr 15, 2020

@hinshun Could you please take a peak at this issue?

stevvooe on Oct 3, 2017