serving: Revision: cannot use private registry

In what area(s)?

/area autoscale

What version of Knative?

0.0.6

Expected Behavior

I should be able to use a private registry.

Actual Behavior

Following https://knative.dev/docs/serving/samples/build-private-repo-go/ I added the secret in the service account but still not able to pull the private image (working fine with a simple deployment) Error in the events:

DENIED: Permission denied for "helloworld" from request "/v2/.../helloworld/manifests/helloworldexample".

Error in the revision:

- lastTransitionTime: "2019-06-20T06:17:04Z"
    message: 'Unable to fetch image ".../helloworld:helloworldexample":
      DENIED: Permission denied for "helloworldexample" from request "/v2/.../helloworld/manifests/helloworldexample". '
    reason: ContainerMissing
    status: "False"
    type: Ready

Steps to Reproduce the Problem

I guess you can try to create a service with an image stored in a private registry.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 21 (14 by maintainers)

Most upvoted comments

Alright, so we figured it out.

Kubernetes supports a special non-standard form of credential in their keyring, which allows credentials to be selected based on a repository (e.g. asia.gcr.io/foo), not just a registry (e.g. asia.gcr.io).

I believe this is because imagePullSecrets are specified at one or both of the Pod or ServiceAccount level (vs. the Container level), which means that you could end up needing multiple distinct credentials for the same registry for different images in the same pod.

e.g. gcr.io/foo/bar:baz and it’s sidecar gcr.io/blah/bloop:bleep might need different credentials.

However, the standard keychain dispatches on the registry hostname.

When I originally wrote k8schain to bridge the K8s keychain and the ggcr keychain concept, I used this little hack to get around this, which works perfectly for well-formed (in the docker sense) keychains: https://github.com/google/go-containerregistry/blob/ca8b66cb1b793f7fb0e3c191ad9a37eb3dc80ea0/pkg/authn/k8schain/k8schain.go#L159

However, for keychains taking advantage of this niche K8s feature, things clearly do not work.

The simple workaround here is to change the key (within the base64 encoded secret blob) from "asia.gcr.io/blah": {...} to "asia.gcr.io": {...}, which for @bappr seems to have done the trick, but we should find a way to squash this inconsistency to keep it from biting other folks.

mattmoor on Jun 27, 2019

Also worth considering adding a more verbose error message around the tag resolver, like:

Error occurred while resolving image digest in knative-serving controller: […]

Otherwise it’s really difficult for an end-user to debug why it’s failing (despite image pull secrets are configured). In one case, we ended up spending almost an hour trying to understand why image pull secret isn’t working.

ahmetb on Sep 5, 2019