serving: Kind + private GCR + Knative Revisions = UNAUTHORIZED

Knative Version: 0.12.0

/area API /area autoscale /area networking

Expected Behavior

Knative Serving (Rest API Example) should be able to pull from a private Google Container Registry (GCR) when using a GCP Service Account to authorize with GCR on kind.

Actual Behavior

Even tho normal Pods and even Knative Pods can use the GCR registry, Knative Revisions cannot.

kubectl get ksvc stock-service-example --output yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: stock-service-example
  namespace: default
...
status:
  conditions:
  - lastTransitionTime: "2020-02-19T00:50:26Z"
    message: 'Revision "stock-service-example-5wlbf" failed with message: Unable to
      fetch image "gcr.io/<redacted>/rest-api-go": failed to resolve image
      to digest: failed to fetch image information: UNAUTHORIZED: You don''t have
      the needed permissions to perform this operation, and you may have invalid credentials.
      To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication.'
    reason: RevisionFailed
    status: "False"
    type: ConfigurationsReady

My assumption is that the Knative Revision is not using the same docker auth config as the kubelet. But I’m not sure if it’s a Knative issue, a Kubernetes issues, or a Kind issue.

Steps to Reproduce the Problem

1. Install kind (from master)

mkdir ~/workspace
cd ~/workspace
git clone git@github.com:kubernetes-sigs/kind.git
cd kind
make build
sudo cp ./bin/kind /usr/local/bin/

1. Configure kind with bind mount for docker config (per private registries doc):

mkdir $HOME/.config/kind/
cat > $HOME/.config/kind/cluster.yaml << EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  # mount docker config that can pull from private GCR
  extraMounts:
  - containerPath: /var/lib/kubelet/config.json
    hostPath: $HOME/.config/kind/docker-config.json
EOF

1. Create GCP service account

REGISTRY=gcr.io/<redacted>
SA_NAME=kubernetes-kind
PROJECT_ID=<redacted>
SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

# Set project to own the service account
gcloud config set project "${PROJECT_ID}"

# Create GCP service account
gcloud iam service-accounts --format='value(email)' create "${SA_NAME}"

# Grant service account permission to read from GCR
gsutil iam ch "serviceAccount:${SA_EMAIL}:objectViewer" "gs://artifacts.${REGISTRY}.appspot.com"

1. Create Docker Auth Config (Ubuntu)

DOCKER_CONFIG=$(mktemp -d)
export DOCKER_CONFIG

# Generate SA Credentials
gcloud iam service-accounts keys create "${DOCKER_CONFIG}/key.json" --iam-account "${SA_EMAIL}"

# Log into GCR
cat "${DOCKER_CONFIG}/key.json" | docker login -u _json_key --password-stdin https://gcr.io

# Copy config to kind bind-mount path
cp "${DOCKER_CONFIG}/config.json" "$HOME/.config/kind/docker-config.json"

rm -r "${DOCKER_CONFIG}"
unset DOCKER_CONFIG

1. Create Docker Auth Config (Mac)

DOCKER_CONFIG=$(mktemp -d)

# Generate Credentials
gcloud iam service-accounts keys create "${DOCKER_CONFIG}/key.json" --iam-account=$SA_EMAIL

# Generate Docker Config
cat > $HOME/.config/kind/docker-config.json << EOF
{
  "auths": {
    "gcr.io": {
      "auth": "$((echo -n "_json_key:" && cat "${DOCKER_CONFIG}/key.json") | base64)"
    }
  },
  "HttpHeaders": {
    "User-Agent": "Docker-Client/19.03.5 (linux)"
  }
}
EOF

rm -r $DOCKER_CONFIG
unset DOCKER_CONFIG

1. Create kind cluster

kind create cluster --config $HOME/.config/kind/cluster.yaml

1. Install Istio 1.4.3 (profile=demo)

cd ~/workspace
curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.4.3 sh -
export PATH="$PATH:$HOME/workspace/istio-1.4.3/bin"
istioctl manifest apply --set profile=demo
kubectl wait pods --all -n istio-system --for=condition=Ready --timeout 2m

1. Install Knative 1.12.0

kubectl apply --selector knative.dev/crd-install=true \
--filename https://github.com/knative/serving/releases/download/v0.12.0/serving.yaml \
--filename https://github.com/knative/eventing/releases/download/v0.12.0/eventing.yaml \
--filename https://github.com/knative/serving/releases/download/v0.12.0/monitoring.yaml
kubectl apply \
--filename https://github.com/knative/serving/releases/download/v0.12.0/serving.yaml \
--filename https://github.com/knative/eventing/releases/download/v0.12.0/eventing.yaml \
--filename https://github.com/knative/serving/releases/download/v0.12.0/monitoring.yaml
kubectl wait pods --all -n knative-serving --for=condition=Ready --timeout 2m
kubectl wait pods --all -n knative-eventing --for=condition=Ready --timeout 2m
kubectl wait pods --all -n knative-monitoring --for=condition=Ready --timeout 2m

1. Install Rest API Example:

cd ~/workspace
git clone -b "release-0.12" https://github.com/knative/docs knative-docs
cd knative-docs/
gcloud auth login
gcloud auth configure-docker
export REPO=gcr.io/<redacted>
docker build \
  --tag "${REPO}/rest-api-go" \
  --file docs/serving/samples/rest-api-go/Dockerfile .
docker push "${REPO}/rest-api-go"
envsubst \
< docs/serving/samples/rest-api-go/sample-template.yaml \
> docs/serving/samples/rest-api-go/sample.yaml
kubectl apply --filename docs/serving/samples/rest-api-go/sample.yaml

1. Check the service/revision conditions

kubectl describe ksvc stock-service-example
...
Status:
  Conditions:
    Last Transition Time:        2020-02-19T00:50:26Z
    Message:                     Revision "stock-service-example-5wlbf" failed with message: Unable to fetch image "gcr.io/<redacted>/rest-api-go:latest": failed to resolve image to digest: failed to fetch image information: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication.
    Reason:                      RevisionFailed
    Status:                      False
    Type:                        ConfigurationsReady