distribution: Mirror not clearing expired entries

Hi,

I am running a docker registry in mirror mode and even though the documentation states that the mirror should periodically remove old content, the disk usage is just growing. Even running a garbage-collect command doesn’t delete anything.

So I looked at some of our images which uses just a single tag (latest) and is forcefully rebuilt every weekend. The repo seems to contain all the builds made since the server was set up:

root@docker-mirror:~# ls -l /srv/docker-mirror/docker/registry/v2/repositories/samepagelabs/debuild/_manifests/tags/
total 4
drwxr-xr-x 4 root root 4096 Jul 25 15:27 latest

root@docker-mirror:~# cat /srv/docker-mirror/docker/registry/v2/repositories/samepagelabs/debuild/_manifests/tags/latest/current/link
sha256:6cd609db8fff28ff3e445fb326dafe745419ab01743a88412474b84004be7216

root@docker-mirror:~# ls -l /srv/docker-mirror/docker/registry/v2/repositories/samepagelabs/debuild/_manifests/tags/latest/index/sha256/
total 60
drwxr-xr-x 2 root root 4096 Aug 15 06:02 055b72b683ad64f8f905eb14d08aa34e454d8153c3bb893286415d6b08489595
drwxr-xr-x 2 root root 4096 Jul 25 15:27 178527a9e52318cd2683f9a938c53fd7bf304ba355eafcb1379a011f7013d265
drwxr-xr-x 2 root root 4096 Sep 21 12:23 201ed34fb2d5e4f8e832fff36e0297ddc3d801f89048c141623958530a308d07
drwxr-xr-x 2 root root 4096 Sep 18 23:49 38482e52778cc7008f9b6e47c657cb04827e0231e6d8212409fbe80b40460270
drwxr-xr-x 2 root root 4096 Aug 29 07:25 50008f7be49dcf01761c5dc7b92b3bdc2eae66f89d8f95599318c7ae0554f44e
drwxr-xr-x 2 root root 4096 Aug  1 05:52 6803f988035a772cb269f79129b6711f651bbd6302ad6c5b9483bc1cbcc98c8d
drwxr-xr-x 2 root root 4096 Sep 25 23:48 6bd01e406e99df1699444a2a3e841d28c8f486ff9cdb8afc05112d9084f05e93
drwxr-xr-x 2 root root 4096 Oct  3 06:44 6cd609db8fff28ff3e445fb326dafe745419ab01743a88412474b84004be7216
drwxr-xr-x 2 root root 4096 Sep  5 06:47 81ccb4aeef5c9606c5ad76b6c7d55bfe3316f36b11705dfac1d3835c588db72a
drwxr-xr-x 2 root root 4096 Aug 22 06:18 9180f131857716a1149be2a7fb8eb2a438c16202d70bfbd092a0e014513133bf
drwxr-xr-x 2 root root 4096 Sep 29 10:50 94b38f2250a85807bd36085e6fc46ed578db9c51b8a1669c08a081b450425099
drwxr-xr-x 2 root root 4096 Aug  8 05:41 a6551cb16e1d4072f4537668176df860e2122723fb21e2f0d6be052dad2ddf36
drwxr-xr-x 2 root root 4096 Oct  2 23:47 ac6c40b569b7f3623bda607193131afef94cbfc8bb39751fc13619f7374c0ab6
drwxr-xr-x 2 root root 4096 Sep 26 09:12 e865a1391e722449ab2791f8180db0ed4dff29a931df4fa03ef635afe84ffc45
drwxr-xr-x 2 root root 4096 Sep 11 23:49 fe61574a5c806441fe65e7aead0ddc12d6b5cd08702489468b35d0911cb57c07

I don’t fully understand the data structures but it seems to me that there are untagged manifests - is that correct? I’d expect them to be deleted.

That untagged manifests are accessible via API:

wget --no-check-certificate https://localhost/v2/samepagelabs/debuild/manifests/sha256:201ed34fb2d5e4f8e832fff36e0297ddc3d801f89048c141623958530a308d07 -O-
--2016-10-05 08:57:32--  https://localhost/v2/samepagelabs/debuild/manifests/sha256:201ed34fb2d5e4f8e832fff36e0297ddc3d801f89048c141623958530a308d07
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:443... connected.
WARNING: no certificate subject alternative name matches
    requested host name 'localhost'.
HTTP request sent, awaiting response... 200 OK
Length: 3028 (3.0K) [application/vnd.docker.distribution.manifest.v2+json]
Saving to: 'STDOUT'

-                                                                      0%[                                                                                                                                                                      ]       0  --.-KB/s               {
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 5762,
      "digest": "sha256:5eee63258bb65a2757bbca610075e39321f7fc5282b23d42e8bfeaf025857f2d"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 51354569,
         "digest": "sha256:2f5fbf61137445d75e8077a9ac5b9b89a2b8eda2dc7486ef42c93da4c5d8760b"
      },
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 91,
         "digest": "sha256:c7401fdb26a3fe4000538d6388d9d10ad70893fb19f0945350615e906064303c"
      },

.....

}

I’ve also noticed the scheduler-state.json file and found that all the entries have expiration date - no date is in the past, all dates are spread over the next week which leads me to assumption, that the expiration period is one week (although I was unable to find this in the documentation 😃). This file doesn’t contain the untagged manifests, just the one tagged as latest, so my next assumption is that some kind of cleaning is actually happening but doesn’t delete files on the disk.

The registry is run via following command:

docker run --name mirror -v /srv/docker-mirror:/var/lib/registry -v /etc/registry/config.yml:/etc/docker/registry/config.yml --rm registry:2

root@docker-mirror:~# docker exec mirror registry --version
registry github.com/docker/distribution v2.5.1

And here’s the configuration file:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

proxy:
  remoteurl: https://registry-1.docker.io
  username: samepageict
  password: *****