distribution: garbage-collect doesn't release much space

Hello, I am having a private docker registry, and we daily pushed images there, I had an job to run garbage-collect periodically, but I found the disk usage still keep increasing.

For example I have one image which is about 1.8G. Before running the garbage-collect the disk usage is

/dev/xvdf1       99G   87G  6.7G  93% /var/vcap/store

After running garbage-collect, the disk usage is

/dev/xvdf1       99G   87G  7.2G  93% /var/vcap/store

Which means after deleting the image, it only releases about 0.5G disk space, so what’s the remaining 1.2G about? The registry version is ./registry github.com/docker/distribution v2.4.1+unknown

When running the garbage-collect I do see below logs like

INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/03/032c9cf1156abb7b4874b8e4023cf4914344d563cacae591dcf883035451f9bb  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/52/526200adb4e562d78fc42fa8db455f82e7ba7cfbb6ca206b9819b1d8cae1b20c  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/56/56652bc41678c3a0c39d71a7b5abfff52424dddcc9b5e10e6ab77f704d54fcdf  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/99/99647b2457ff2c14d0021fdda1136803b25d57d008957bdd5a18cd629c5b4e85  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/ce/ce7242dfaaec621f029795149ab0d6cb5467c5dc147680a21c6bda0064bb4a00  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/e3/e3a3b148f89d3c5ec415b0310936ad88afd814b5da2507a8c2e898d779f0174d  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/10/103dacd4981428e62a92586e434f31f021a84de6a3906e6cfca9b1f2c8bb77df  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/11/116679ed92079caa183bc74a6300b514b29ac5d6a086fbc9a1c3d1fb18d8f0f2  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/17/179bd293d2dfc0c039891e50ac056339f493fcca0b0fb4795fb67ef6ffe86efc  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/4f/4f4862a850ac45e9eca81b6dc4ca1cedc91c79e5316f125b4bb4b0b14e47d1af  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/89/8910ed079727b2440fe714e973181c624e688d09724b93f3a6576d4f426ff8c1  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/ad/adcaecf0c3a2c74b7d90a9b22ed8be10d2486723e6c4686430ac16381efbebe0  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/b1/b16bf7888d5ffeefad15ec2a811f247e65b6c75d4f79fa21acf14ce187594755  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/d9/d95803f917b8505a6a0a4c98c4f8710381dfa0b1f0c4f372ba20906c5f9c9995  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/18/18aefb43df71fc073c41a04689db8132ca5d803a50e8567ee5fff0297015e564  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/19/1983d3edccb0aead4e3fb3f7e5d8f38e85f8b74b8bcad1981fc3380c2e04478f  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/8a/8a13effafcb0396b401a116d0553f2ceeaedf0b934ee838a7c99cafdf9804fa0  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/92/9226016f37365aae77f30e37d47aa9d5934f7a8204eeceeef5704ef0dd9f3dd9  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/b3/b3741efd03d4535eeb167fdd6a7765aeca9dfcbd489a8a631da7177ae9b3ebc4  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/d4/d4e191bd4283c8ddc4cc57d9edf987a5cf773fb59591d993a54802f8ed59467c  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/5b/5b6df681cb1f4d11e6d1acf08d7dcfbed4c20bbcfec916f189b0946959332548  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/60/603a6affdfcb6943bc53535db3a3ddf376a1c7d7289c8418827b7e3ebc9a1c67  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/64/64e73dd0f364efef29b9abea57b59003d08b2317a8b1e440b3802db46b5c93dc  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry
INFO[0002] Deleting blob: /docker/registry/v2/blobs/sha256/ca/ca0ec1eac59887dc3438fc175fc9330258764d64016fc983d36866119832f7aa  environment=registry-0 go.version=go1.7 instance.id=34dac3e9-0a18-4646-8650-15131f0d535a service=registry

Is there any known issue about this?

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 31

Most upvoted comments

Ok found it, firstly, we used https instead of http, so it did http://https😕/… . Secondly, I added --insecure in curl, because we use auto-generated certificate for now. We reclaimed ~40GB of space, that’s cool!

Somehow, this procedure should be in the official doc one day, because the workflow of overwriting latest layers in very used in every development environment.

Thank you again.

antoinetran on Feb 13, 2017

Hi,

I have now created a bit more sustainable solution for this. Please use/star/ignore as you see fit.

https://github.com/mortensteenrasmussen/docker-registry-manifest-cleanup https://hub.docker.com/r/mortensrasmussen/docker-registry-manifest-cleanup

mortensteenrasmussen on Feb 19, 2017

Hi, it sounds like you might have a lot of untagged manifests. To verify this, could you please run this on your registry server in the $REGISTRY_DIR/data/docker/registry/v2/repositories folder.

It will return the number of manifests you have in your registry that doesn’t have a tag associated.

comm -23 <(find . -type f -name "link" | grep "_manifests/revisions/sha256" | grep -v "\/signatures\/sha256\/" | awk -F/ '{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep "_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g'; echo; done | sort) | wc -l

The command looks cryptic, but it uses the filesystem to check tags vs. manifests and returns the manifests that don’t match any tags. If this number is high, I can help you with a command that can delete them. I’m planning to put it on Github, but I’m still testing/verifying it.

mortensteenrasmussen on Feb 13, 2017

We deployed a cron that executes this script. But we encountered an issue that may be related to this script. How well does this script resist to race condition? The symptoms are:

sudo docker pull [Image]:[Version]
[Version]: Pulling from ...
45a2e645736c: Already exists
fffdc9cd00d0: Already exists
991e46ec94a9: Already exists
3539362589ad: Already exists
65dd1f11758e: Already exists
9e3ed43117dc: Already exists
7d6432f472af: Already exists
74c725b2370d: Already exists
97246cd202b6: Already exists
9b91802bdd44: Already exists
75f47dcffa4f: Already exists
ad1062d1ad56: Already exists
69f1b2e5b5c8: Already exists
fa42d3d5997a: Already exists
e1f5e167f582: Already exists
039ddd2e6b7a: Already exists
f4fe203c0bcc: Already exists
cd5ab8a2f539: Already exists
1b93c8b04a78: Already exists
50dda98c3ed4: Already exists
353b97c92a07: Already exists
1b7803fc3124: Retrying in 4 seconds
49eb1480916e: Retrying in 4 seconds
b2c65301a004: Retrying in 4 seconds
690f6c2b6a71: Waiting
f6f72b43ce9a: Waiting
aa218eca26b2: Waiting
f1eafb397587: Waiting

sudo docker pull [Image]:[Version]
[Version]: Pulling from ...
45a2e645736c: Already exists
fffdc9cd00d0: Already exists
991e46ec94a9: Already exists
3539362589ad: Already exists
65dd1f11758e: Already exists
9e3ed43117dc: Already exists
7d6432f472af: Already exists
74c725b2370d: Already exists
97246cd202b6: Pull complete
9b91802bdd44: Pull complete
75f47dcffa4f: Pull complete
ad1062d1ad56: Pull complete
69f1b2e5b5c8: Pull complete
fa42d3d5997a: Pull complete
e1f5e167f582: Pull complete
039ddd2e6b7a: Pull complete
f4fe203c0bcc: Pull complete
cd5ab8a2f539: Pull complete
1b93c8b04a78: Pull complete
50dda98c3ed4: Pull complete
353b97c92a07: Pull complete
1b7803fc3124: Downloading [==================================================>]  1.94 kB/1.94 kB
49eb1480916e: Downloading [==================================================>]   859 MB/859 MB
b2c65301a004: Downloading [==================================================>] 2.434 kB/2.434 kB
690f6c2b6a71: Download complete
f6f72b43ce9a: Downloading [==================================================>]    691 B/691 B
aa218eca26b2: Downloading [==================================================>]    704 B/704 B
f1eafb397587: Downloading [==================================================>]    216 B/216 B
unexpected EOF

We couldn’t pull an image. Even rebuilding the image does not fix. We had to delete the image directory (with all tags/revisions) ([RegistryDir]/[RepoName]/[ImageName]) and garbage-collect, then rebuild the image, for the pull to work. I believe some layers was missing because they were removed by the cron script, while another image was being pushed. I also suspect this because our logs showed that a clean and a push was processing at the same time.

Anyone has seen this?

antoinetran on Feb 24, 2017

Hi, we have the exact same issue. We followed the procedure the official doc, but end up still having full space. In our case, we have images that has no tag at all, and it still takes lots of space. Only when we delete the hole image (that contains no tag), then we can garbage-collect a lot of space.

Here is the result of the command:

comm -23 <(find . -type f -name "link" | grep "_manifests/revisions/sha256" | grep -v "\/signatures\/sha256\/" | awk -F/ '{print $(NF-1)}' | sort) <(for f in $(find . -type f -name "link" | grep "_manifests/tags/.*/current/link"); do cat ${f} | sed 's/^sha256://g'; echo; done | sort) | wc -l
299

I was suspicious the manifest still has a lots of references to unused layers, even when we delete tags.

antoinetran on Feb 13, 2017