skopeo: Sync is very, very slow.
Hello,
Does anybody have any thoughts on how to speed up a sync. We mirror a lot of external repos on mass for all versions and the process can take several hours layer by layer, tag by tag, repo by repo.
Is there anyway to parallel to run for every repo ? I’ve thought about maybe splitting each external repo in to separate yamls and run more skopeo commands at same time.
skopeo --insecure-policy sync --src yaml --dest docker src_mirrors.yaml dest.internal.repo/mirrors/ --scoped --dest-creds X:X
docker.io:
images:
istio/proxyv2: []
sergrua/kube-tagger: []
splunk/fluentd-hec: []
hazelcast/management-center: []
k8srestdev/scaling: []
hjacobs/kube-resource-report: []
tutum/dnsutils: []
bitnami/external-dns: []
falcosecurity/falco: []
praqma/helmsman: []
weaveworks/flagger: []
weaveworks/flagger-loadtester: []
quay.io:
images:
coreos/prometheus-config-reloader: []
prometheus/prometheus: []
prometheus/alertmanager: []
prometheus/node-exporter: []
kiali/kiali: []
coreos/configmap-reload: []
coreos/prometheus-operator: []
thanos/thanos: []
calico/node: []
coreos/kube-state-metrics: []
calico/typha: []
external_storage/efs-provisioner: []
gcr.io:
images:
kubernetes-helm/tiller: []
google_containers/busybox: []
k8s-artifacts-prod/autoscaling: []
k8s.gcr.io:
images:
cluster-proportional-autoscaler-amd64: []
metrics-server-amd64: []
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 4
- Comments: 23 (8 by maintainers)
Some good news - https://github.com/containers/image/pull/1041 reduces the sync time of about 650 images from ~35mins to ~7mins (!)
With roughly the simply change you described in
containers/image/copy.copyOneImageSo this seems like a nice performance win for the use case of synchronizing a large number of images which are expected to already exist.
The PR is still very much a draft, but this functionality would certainly be a good to have IMHO for this usage pattern. If you would like to consider please @mtrmac
Thanks, that’s very encouraging. I’m not sure I’ll have the time to look into the details of the behavior this week, but it’s definitely an important improvement to prioritize.
Ouch. Yeah, in the best possible case reading 4000 manifests to see whether the tags have moved (and that can only be done one tag at a time) is going to take some time. OTOH if most of the time is spent waiting, parallelizing this across a few threads might noticeably help. The code structure right now makes that a bit difficult (the checks are part of a copy, and a copy automatically parallelizes copies of individual layers, so a dozen copies * 6 layer threads = 70 parallel TCP streams), it might make sense to do a “presence” check before starting a copy if almost all images are not added/updated.
@samdoran and I forgot to add, TYVM for sharing the results!
I can confirm with
skopeo1.2.3 sycing is much faster because nothing is copied if it already exists in the destination. Our dailyskopeo syncwent from ~20 minutes to less than five minutes. 🎊That would be interesting. Very roughly, in
containers/image/copy.copyOneImage, somewhere afterIsRunningImageAllowed(exact placement TBD, probably affects correctness!, but this could be good enough for a basic performance estimate), compare that the byte arrays returned byunparsedImage.Manifest()andc.dest.Reference().NewImageSource().GetManifest()(+ appropriate error checking on all the calls), and claim success if they are equal.The “Copying blob/Writing manifest” entries record starts of that activity, and we do up to 6 blob copies in parallel. So, going by this specific record, if I’m reading it right, reading the original manifest and config took 0 ms, ??? set up took 260 ms, checks that the layers are present took 523 ms, writing the config 263 ms, writing the manifest 0 ms. Several of those times just don’t make sense to me; but it’s certainly true that the layer checks do take a big part of that.
Yes, copies are implemented in a way that assumes the destination needs to be updated; an extra “is the destination already exactly what we want it to be” check would, for most copies, be just an extra delay. OTOH for sync it’s probably the other way around, and it may well make sense to opt in to doing that check (maybe just assuming that the image does not need converting). Or, in the extreme, don’t ask the destination at all, and maintain a local state file that records the last manifest digest written to a particular tag. (We would probably still have to query each individual tag at the source, all 4000 or however many of them.)
This sync thing is starting to grow fairly complex…
I have the same sort of problem with sync (which is very nice by the way!). The use case is to ensure ‘remote’ registries are in sync with an internal/source registry. And we run a periodic job which ‘duplicates’ out internal image/tags. But where generally the majority of the content is unmodified.
The below timestamps are for a specific image copy (where the destination image already exists). Which took just over 1s. This is 11 minutes or so for the 685 tags (and there are a few more images).
From looking at the source, it seems
syncimplements a ‘copy’ for each tag (reasonable ,of course). But in case where the destination is expected to exist unmodified, are there any possible optimizations? In particular the writing of the manifest seems to always take ~300ms, but should be unchanged?