origin: Cannot pull from istag referencing image in another repository of internal registry

Reproducer:

$ docker tag docker.io/hello-world 172.30.104.152:5000/tmp/hello-world:latest
$ docker push 172.30.104.152:5000/tmp/hello-world:latest
The push refers to a repository [172.30.104.152:5000/tmp/hello-world] (len: 1)
0a6ba66e537a: Pushed 
b901d36b6f2f: Pushed 
latest: digest: sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1 size: 2748

$ oc tag -n tmp hello-world@sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1 mybase:latest
Tag mybase:latest set to hello-world@sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1.

*$ oc describe -n tmp istag
Name:           sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1
Created:        2 minutes ago
Labels:         <none>
Annotations:    openshift.io/image.managed=true
Docker Image:   172.30.104.152:5000/tmp/hello-world@sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1
Image Name:     sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1
Image Size:     633 B (first layer 601 B, last binary layer 32 B)
Image Created:  4 months ago
Author:         <none>
Arch:           amd64
Command:        /hello
Working Dir:    <none>
User:           <none>
Exposes Ports:  <none>
Docker Labels:  <none>


Name:           sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1
Created:        2 minutes ago
Labels:         <none>
Annotations:    openshift.io/image.managed=true
Docker Image:   172.30.104.152:5000/tmp/hello-world@sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1
Image Name:     sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1
Image Size:     633 B (first layer 601 B, last binary layer 32 B)
Image Created:  4 months ago
Author:         <none>
Arch:           amd64
Command:        /hello
Working Dir:    <none>
User:           <none>
Exposes Ports:  <none>
Docker Labels:  <none>

$ # note the first pull is successful because all the blobs are already downloaded in Docker daemon's storage
$ docker pull 172.30.104.152:5000/tmp/mybase:latest                                                                                                                                                                                           
Trying to pull repository 172.30.104.152:5000/tmp/mybase ... latest: Pulling from tmp/mybase
Digest: sha256:bb5c16ecd58633d4088140acb6ee386c5c3b09a8f03c517a288093415f0c09d1
Status: Downloaded newer image for 172.30.104.152:5000/tmp/mybase:latest

$ docker rmi -f 0a6ba66e537a
Untagged: 172.30.104.152:5000/tmp/hello-world:latest
Untagged: 172.30.104.152:5000/tmp/mybase:latest
Untagged: docker.io/hello-world:latest
Deleted: 0a6ba66e537a53a5ea94f7c6a99c534c6adb12e3ed09326d4bf3b38f7c3ba4e7
Deleted: b901d36b6f2fd759c362819c595301ca981622654e7ea8a1aac893fbd2740b4c
$ # the second pull needs to repull all the blobs and fails
$ docker pull 172.30.104.152:5000/tmp/mybase:latest
Trying to pull repository 172.30.104.152:5000/tmp/mybase ... latest: Pulling from tmp/mybase
b901d36b6f2f: Pulling fs layer 
0a6ba66e537a: Pulling fs layer 
not found
Error: image tmp/mybase:latest not found

Tested with Docker 1.8.2 and 1.9.1. Test to confirm this is in #7790.

See also https://github.com/openshift/origin/pull/7128#issuecomment-191754538 and https://github.com/openshift/origin/pull/7763#issuecomment-191968269 for more background.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 44 (34 by maintainers)

Commits related to this issue

Most upvoted comments

Here’s what we’re pursuing right now:

We’re adding a new middleware wrapper (e.g. localMultiplexerBlobStore) that will multiplex between local repositories in order to serve requested blob living in some other repository referenced in corresponding image stream.

The resulting blob store will be a composition of:

pullghroughBlobStore{ localMultiplexerBlobStore{ quotaRestrictedBlobStore{ upstreamBlobStore } } }

It will:

  • implement Stat and ServerBlob methods
  • behave like this:
    1. pass the request down to the quotaRestictedBlosStore
    2. if not found, it will determine, whether the blob is linked in any other local repository referenced by currently requested IS; user must be authorized to access the target repository user must be authorized only to access imagestreams/layers of the IS he’s pulling from, he doesn’t need the permissions on the target repository
    3. it will instantiate a repo object for the repository and pass the request to it

What about caching? Two approaches:

  1. map blob digests ➡️ found local repository names
    • subsequent requests will hit the cache
  2. link the layers of the found image in current repository
    • subsequent requests will be handled by the underlying blob store
    • there’s a drawback that layers will stay linked even after the istags to other local repositories are deleted; this will be partially addressed by reworked pruning (blobs will stay linked until a periodic garbage collection runs)
+1
miminar on Mar 21, 2016

Well, let me say then “this is a regression from the design of the core feature of the product” and “if we had detected it at any point along the way it would have been an immediate P0” 😃

On Fri, Mar 18, 2016 at 10:20 AM, Michal Minar notifications@github.com wrote:

So prior to us adding pullthrough, this worked.

@legionus https://github.com/legionus confirmed this is an issue even with origin v1.0.1 so it seems to me like a very old bug rather than a regression

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/openshift/origin/issues/7792#issuecomment-198380545

+1
smarterclayton on Mar 18, 2016
Read more comments on GitHub
← origin: Cannot fetch from a self-signed HTTPS git repository
origin: Cannot specify integer/bool/base64 types in templates →