skopeo: Skopeo sync fails with 400 (Bad Request) on Quay.io (deeply nested repo name + bad error handling)

Hello,

I’m trying to sync some images from docker.io to my private registry, but am seeing a strange issue that I am confused by:

skopeo sync --src docker --dest docker docker.io/library/ubuntu:latest registry.local.com/drew/ubuntu --override-os linux --debug

Output

INFO[0000] Tag presence check                            imagename="docker.io/library/ubuntu:latest" tagged=true
DEBU[0000] Destination for transport "docker": //registry.local.com/drew/ubuntu/ubuntu:latest
INFO[0000] Copying image ref 1/1                         from="docker://ubuntu:latest" to="docker://registry.local.com/drew/ubuntu/ubuntu:latest"
DEBU[0000] Returning credentials from /Users/drew/.config/containers/auth.json
DEBU[0000] Using registries.d directory /usr/local/etc/containers/registries.d for sigstore configuration
DEBU[0000]  Using "default-docker" configuration
DEBU[0000]   Using file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.local.com
DEBU[0000] Loading registries configuration "/usr/local/etc/containers/registries.conf"
DEBU[0000] Trying to access "docker.io/library/ubuntu:latest"
DEBU[0000] Credentials not found
DEBU[0000] Using registries.d directory /usr/local/etc/containers/registries.d for sigstore configuration
DEBU[0000]  Using "default-docker" configuration
DEBU[0000]  No signature storage configuration found for docker.io/library/ubuntu:latest, using built-in default file:///Users/drew/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fubuntu%3Apull&service=registry.docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/library/ubuntu/manifests/latest
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0000] Using blob info cache at /Users/drew/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] Source is a manifest list; copying (only) instance sha256:3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e for current system
DEBU[0000] GET https://registry-1.docker.io/v2/library/ubuntu/manifests/sha256:3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0000] IsRunningImageAllowed for image docker:docker.io/library/ubuntu:latest
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Reading /Users/drew/.local/share/containers/sigstore/library/ubuntu@sha256=3093096ee188f8ff4531949b8f6115af4747ec1c58858c091c8cb4579c39cc4e/signature-1
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking /v2/drew/ubuntu/ubuntu/blobs/sha256:f611acd52c6cad803b06b5ba932e4aabd0f2d0d5a4d050c81de2832fcb781274
DEBU[0000] GET https://registry.local.com/v2/
DEBU[0000] Checking /v2/drew/ubuntu/ubuntu/blobs/sha256:83ee3a23efb7c75849515a6d46551c608b255d8402a4d3753752b88e0dc188fa
DEBU[0000] Checking /v2/drew/ubuntu/ubuntu/blobs/sha256:db98fc6f11f08950985a203e07755c3262c680d00084f601e7304b768c83b3b1
DEBU[0001] Ping https://registry.local.com/v2/ status 401
DEBU[0001] GET https://registry.local.com/v2/auth?account=drew&scope=repository%3Adrew%2Fubuntu%2Fubuntu%3Apull%2Cpush&service=registry.local.com
DEBU[0001] GET https://registry.local.com/v2/auth?account=drew&scope=repository%3Adrew%2Fubuntu%2Fubuntu%3Apull%2Cpush&service=registry.local.com
DEBU[0001] GET https://registry.local.com/v2/auth?account=drew&scope=repository%3Adrew%2Fubuntu%2Fubuntu%3Apull%2Cpush&service=registry.local.com
FATA[0001] Error copying ref "docker://ubuntu:latest": Error trying to reuse blob sha256:83ee3a23efb7c75849515a6d46551c608b255d8402a4d3753752b88e0dc188fa at destination: Requesting bear token: invalid status code from registry 400 (Bad Request)

I have full access to push to the private registry. I do so on a regular basis via docker push. It is a Quay registry, and I have made sure that docker login succeeds prior to sync/push.

Have also just shortened the source path:

skopeo sync --src docker --dest docker ubuntu:latest registry.local.com/drew/ubuntu --override-os linux --debug

I’ve tried adding --dest-creds and the flags to ignore TLS just as a test. No bueno.

It seems a skopeo copy does work… which I find odd.

skopeo copy docker://ubuntu:latest docker://registry.local.com/drew/ubuntu:latest --override-os linux

Copy works… sync doesn’t…

How can I diagnose what’s going on here?

skopeo v1.2.1 Source: DockerHub (docker.io) Destination: Quay v3.3.4 (private registry, registry.local.com)