pipeline: Missing permissions while downloading bundles from Google Artifact Registry using v0.33.2 (was working in v0.32.0)

After upgrading tekton-pipelines-controller from v0.32.0 to v0.33.2 it can’t download bundles from the Google Artifact Registry. We use a separate k8s service account as default for pipelineruns, it has imagepullsecret with .dockerconfigjson inside.

Expected Behavior

tekton-pipelines-controller downloads pipeline definition (bundle) from the Google Artifact Registry and creates pipelinerun (just like v0.32.0 did)

Actual Behavior

Error retrieving pipeline for pipelinerun <namespace>/<name of pipelinerun>: error when listing pipelines for pipelineRun <name of pipelinerun>: GET https://us-east4-docker.pkg.dev/v2/token?scope=repository%3A<gcp-project-name>%2F<repo-path>%3Apull&service=us-east4-docker.pkg.dev: DENIED: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "<full-repo-path>"

Steps to Reproduce the Problem

Create secret with .dockerconfigjson inside with enough permissions to download from the repo
Create serviceaccount and use this secret as ImagePullSecret
Create pipeline and publish it’s bundle (preferably to Google Artifact Registry)
Create pipelinerun using ref to previously published bundle and using previously created serviceaccount
Observe an error in the tekton dashboard/tekton-pipelines-operator logs/pipelinerun status

Additional Info

Kubernetes version:

Output of kubectl version:

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6-gke.1503", GitCommit:"2c7bbda09a9b7ca78db230e099cf90fe901d3df8", GitTreeState:"clean", BuildDate:"2022-02-18T03:17:45Z", GoVersion:"go1.16.9b7", Compiler:"gc", Platform:"linux/amd64"}

Tekton Pipeline version:

Pipeline version: v0.33.2

I suspect this keychain generation, because k8schain was updated between v0.32.0 and v0.33.2 https://github.com/tektoncd/pipeline/blob/6b5710c3242ce7c3e1e8b876b7d35316449f617a/pkg/reconciler/pipelinerun/resources/pipelineref.go#L58 K8schain was upgraded here: https://github.com/tektoncd/pipeline/pull/4488

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 11
Comments: 17 (8 by maintainers)

Most upvoted comments

Thanks for clarifying. So the issue isn’t specific to auth, it’s just that the solution to the auth problem requires setting an annotation on the SA, and the Operator doesn’t support that well. Is that correct?

If so, I’d suggest filing a new issue on the Operator, referencing this issue, covering the fact that SA annotations are overwritten by the Operator.

Does that make sense?

imjasonh on Jul 13, 2022

Right, it’s GKE with Workload Identity enabled.

Oh that’s super interesting! So it sounds like the k8schain change actually just ignored your .dockerconfigjson imagepullsecret (that’s a bug!) and fell back onto the GKE WI SA, which didn’t have the right permissions.

So yes if we add proper annotation to the KSA tekton-pipeline-controller it will work, but there is no way to alter the KSA via Tkn operator config.

That sounds like a reasonable feature request to me. In the meantime, if you add the annotation, does the Operator (or anything else) remove it for you? I think that might be the better way to go in general, since it means you’re not storing registry credentials in a secret.

Yes, it would be much better to just use Workload Identity then generate .dockerconfigjson

mike-serchenia on Mar 17, 2022

Right, it’s GKE with Workload Identity enabled.

Oh that’s super interesting! So it sounds like the k8schain change actually just ignored your .dockerconfigjson imagepullsecret (that’s a bug!) and fell back onto the GKE WI SA, which didn’t have the right permissions.

So yes if we add proper annotation to the KSA tekton-pipeline-controller it will work, but there is no way to alter the KSA via Tkn operator config.

That sounds like a reasonable feature request to me. In the meantime, if you add the annotation, does the Operator (or anything else) remove it for you? I think that might be the better way to go in general, since it means you’re not storing registry credentials in a secret.

imjasonh on Mar 17, 2022

So the error message is a little interesting here, because it sounds like a 403 (you don’t have the right permission), and not a 401 (we don’t know who you are).

k8schain updates might be the cause, but I’d expect that to result in creds not being sent at all, or sent incorrectly, resulting in a 401, not what we’re seeing here.

Just in case it ends up mattering (unlikely), are you running in a GKE cluster, potentially with Workload Identity enabled? That might cause the k8schain change to select a different valid account than under v0.32.0, and the newly selected account might not have the right permissions.

Right, it’s GKE with Workload Identity enabled. So yes if we add proper annotation to the KSA tekton-pipeline-controller it will work, but there is no way to alter the KSA via Tkn operator config.

mike-serchenia on Mar 17, 2022