harbor: [GC performance] The performance of v2 manifest deletion is not good in S3 environment
In S3 backend environment, we found that it took about 39 seconds to delete a manifest via v2 API.
[why still use v2 to handle manifest deletion]
As Harbor cannot know the tags belong to the manifest in the storage, the GC job needs to leverage the v2 API to clean them. But, the v2 API will look up all of tags, and remove them one by one. This may cause performance issue.
[what we can do next]
1, Investigate how many requests send to S3 storage within the v2 manifest deletion. 2, Investigate the possibility of not to store the first tag in the backend, then GC job can skip this step.
Log
Sep 1 12:56:35 192.168.144.1 registry[1146]: time="2020-09-01T12:56:35.750530108Z" level=info msg="authorized request" go.version=go1.13.8 http.request.host="registry:5000" http.request.id=c9a4d5ad-4157-4091-a023-93d8e20a5746 http.request.method=DELETE http.request.remoteaddr="192.168.144.9:44072" http.request.uri="/v2/library/testingg/manifests/sha256:20f39c20df7c5605f77862b711c3d28731e4d569171ec852ce34a06432611faa" http.request.useragent=harbor-registry-client vars.name="library/testingg"
vars.reference="sha256:20f39c20df7c5605f77862b711c3d28731e4d569171ec852ce34a06432611faa"
Sep 1 12:57:14 192.168.144.1 registry[1146]: time="2020-09-01T12:57:14.340710966Z" level=info msg="response completed" go.version=go1.13.8 http.request.host="registry:5000" http.request.id=c9a4d5ad-4157-4091-a023-93d8e20a5746 http.request.method=DELETE http.request.remoteaddr="192.168.144.9:44072" http.request.uri="/v2/library/testingg/manifests/sha256:20f39c20df7c5605f77862b711c3d28731e4d569171ec852ce34a06432611faa" http.request.useragent=harbor-registry-client http.response.duration=38.598453034s http.response.status=202 http.response.written=0
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 13
- Comments: 35 (13 by maintainers)
When using s3 for storage, during manifest delete, s3 look-up using s3 list and get calls is performed to check all the tags referencing this manifest. For a large repository having lakhs of images, all these tags are read to delete 1 single image.
Instead of relying on distribution to delete the tag folder, since Harbor already maintains the tags referencing a manifest, we can introduce a new api in distribution to delete the tag directory altogether when deleting the artifact and skip tag lookup during the garbage collection step.
I was able to test this out and brought down the registry size from 360TB to less than 30TB by configuring 30 days of retention period. Checkout these prs: https://github.com/hemanth132/harbor/pull/1 https://github.com/hemanth132/harbor/pull/2 https://github.com/hemanth132/distribution/pull/2
Have done a couple of optimizations like:
I made the changes against the version 2.7.0 code of Harbor. Now that this is working, will try to raise pr in the main harbor repo in the coming weeks.
Please let me know if this helps for you.
Hello friends, I’d like to ask to raise the priority of this issue.
We are running several instances of Harbor (we use GCS backend, but I think the root cause here is the same) and we’re rapidly growing our usage.
We are starting to reach capacities that the GC simply cannot handle, repositories with more than a few thousand tags is taking ~2 minutes to delete a single manifest during GC, GC is now taking 10~14 hours and the problem is getting worse every day since we’re adding more tags then we are deleting.
on our test/certification Harbor instance we’ve reached over 20,000 tags on some repositories and GC just times out on the first manifest since the lookup takes >20 minutes.
We are concerned about increasing our storage costs since we can’t clean it up as well as other potential issues that may arise from having all these blobs/manifests lingering with no ability to properly clean them up.
this issue was tagged as a candidate for v2.2.0, and we’re already seeing v2.4.0 going out the door.
I’m happy to provide additional context, information, logs but just hope we can have some attention on this issues since I think it will impact any user that needs Harbor to work at scale.
/cc @wy65701436 @reasonerjt
Just to further explain the crux of this issue.
Harbor is using the docker distribution for it’s (harbor-registry) registry component.
The harbor GC will call into the docker registry to delete the manifest, which in turn then does a lookup for all tags that reference the deleted manifest: https://github.com/distribution/distribution/blob/78b9c98c5c31c30d74f9acb7d96f98552f2cf78f/registry/handlers/manifests.go#L536
To find the tag references, the docker registry will iterate every tag in the repository and read it’s link file to check if it matches the deleted manifest (i.e. to see if uses the same sha256 digest): https://github.com/distribution/distribution/blob/78b9c98c5c31c30d74f9acb7d96f98552f2cf78f/registry/storage/tagstore.go#L160
So, the more tags you have in your repository, the worse the performance will be (as there will be more s3 API calls occurring for the tag directory lookups and tag file reads).
@karmicdude This could be a fix for that https://github.com/distribution/distribution/pull/3827 Harbor uses distribution under the hood and I already tested it for my Harbor instance
@dkulchinsky Image is in our private registry, but the instruction is this: Build distrubution bin from my fork via
go build -o bin/registry ./cmd/registryAnd place the bin insideharbor-registryfolder of this slightly modified bitnami repo harbor-registry.tar.gzThen docker build
@wy65701436 any hints on when the team may find some time to look at this? this seems like an issue that requires desperate attention however it didn’t see any traction in over 2 years now.
@Antiarchitect, wanted to share some progress
I was able to build distribution with your PR, https://github.com/distribution/distribution/pull/3702 (update GCS drive to latest) and the Redis sentinel patch.
I also figured out the issue around the retry loop when a 404 is returned during manifest delete and have a fix for Harbor jobservice here: https://github.com/goharbor/harbor/pull/18386
I’m running a test now with the above in our sandbox Harbor and although it’s not breaking any speed records, it is looking much better, will update once I have more concrete numbers.
Yes, we use the minio-operator and minio
Also reading from the disk is stuck at the upper limit of IOPS (320 by current disks specifications).
tenantwas created under harbor registry. Now the drives are networked, with the specifications listed above. You can see that there is a problem, when GC starts, i/o wait ~450ms - a lot.We will test a similar setup on fast disks (ssd or nvme) with the same data made from snapshots to see how the situation changes when GC runs. I will post when I have some results. But it is unlikely to improve the issue much. If the situation gets much better, we always have the option to deploy minio on dedicated server with high-performance local disks.
I believe this is still an active issue being tracked, so probably shouldn’t get closed yet.