harbor: Cannot replicate images from Red Hat registry (registry.access.redhat.com)
Trying to replicate an image from the public Red Hat Container Registry (for instance ubi7/ubi-minimal:7.7-10) fails even though the same image can be pulled by a standard Docker daemon and Red Hat’s own podman.
Expected behavior and actual behavior
Expected
Harbor should pull the image from the registry and add it to a local project.
Actual
The replication job fails with strconv.ParseInt: parsing "": invalid syntax while trying to pull the first blob referenced by the manifest.
Steps to reproduce the problem
- Create a registry with name
redhat-registry, typedocker-registryand URLhttps://registry.access.redhat.com - Create a project (in my case:
ubi7) - Create a replicaiton rule with the following settings:
- Name:
redhat-ubi7 - Replication mode: Pull-based
- Source registry:
redhat-registry - Source resource filter:
- Name:
ubi7/ubi-minimal - Tag: 7.7-10
- Resource: image
- Name:
- Destination namespace:
ubi7 - Trigger Mode: Manual
- Override: Yes
- Enable rule: Yes
- Trigger the replication rule
Versions
- harbor version: [1.9.0]
- docker engine version: [18.09.4]
- docker-compose version: [1.22.0]
Additional context
Relevant excerpt from the job log:
2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:95]: client for source registry [type: docker-registry, URL: https://registry.access.redhat.com, insecure: false] created
2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:105]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:157]: copying ubi7/ubi-init:7.7-10(source registry) to ubi7/ubi-init:7.7-10(destination registry)...
2019-09-23T12:36:21Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image ubi7/ubi-init:7.7-10 ...
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:272]: the manifest of image ubi7/ubi-init:7.7-10 pulled
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:294]: trying abstract a manifest from the manifest list...
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:306]: a manifest(architecture: amd64, os: linux) found, using this one: sha256:c88406b7e200c6a518d594fa5747131c412e38918faf2eb7104e8e1dad641b2c
2019-09-23T12:36:22Z [INFO] [/replication/transfer/image/transfer.go:261]: pulling the manifest of image ubi7/ubi-init:sha256:c88406b7e200c6a518d594fa5747131c412e38918faf2eb7104e8e1dad641b2c ...
2019-09-23T12:36:23Z [INFO] [/replication/transfer/image/transfer.go:272]: the manifest of image ubi7/ubi-init:sha256:c88406b7e200c6a518d594fa5747131c412e38918faf2eb7104e8e1dad641b2c pulled
2019-09-23T12:36:23Z [INFO] [/replication/transfer/image/transfer.go:231]: copying the blob sha256:a99e238f6cc16b6fbd1d613195a50b4ba7bb6358bebad982215807b62331ccb1...
2019-09-23T12:36:23Z [ERROR] [/replication/transfer/image/transfer.go:244]: failed to pulling the blob sha256:a99e238f6cc16b6fbd1d613195a50b4ba7bb6358bebad982215807b62331ccb1: strconv.ParseInt: parsing "": invalid syntax
2019-09-23T12:36:23Z [ERROR] [/replication/transfer/image/transfer.go:143]: strconv.ParseInt: parsing "": invalid syntax
Drilling down from replication/transfer/image/transfer.go to the place where strconv.ParseInt is called when processing blobs, we end up here: https://github.com/goharbor/harbor/blob/cb5a145f275464c5044caa951a3ad99a72a10e1d/src/common/utils/registry/repository.go#L347-L351
Basically, Harbor uses the header to get the blob size instead of parsing it from the manifest, so if the upstream registry does not provide said header it doesn’t pull the image and the replication job fails.
Again, both the upstream Docker daemon and the podman utility can successfully pull images from the Red Hat registry (they probably don’t even bother checking for the Content-Length header since they get the blob size from the manifest), so the absence of Content-Length should not cause an error.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 16 (11 by maintainers)
@rfc1459 @finkr @phisco
We’ll take a look at the
Content-lengthchecking issue. Any progress, I’ll post them here.@steven-zou : I was able to replicate the ubi8minimal image using the authenticated endpoint registry.redhat.io. I was furthermore able to replicate the ubi7minimal image using the unauthenticated endpoint registry.access.redhat.com
The performance was quite low, which might be caused by my testenvironment and unrelated can’t test it through corporate proxy, but am confident (since it was the same error) that this issue is gone too.
ubi8minimal replication:
ubi7minimal replication