velero: Regression: Disable blob signing initialization for non SA file results in GCP Workload Identity not working

What steps did you take and what happened:

I’m upgrading from velero 1.11.1 to 1.12.0 and velero-plugin-for-gcp from 1.7.1 to 1.8.0. The GCP plugin introduced this change - https://github.com/vmware-tanzu/velero-plugin-for-gcp/pull/142 - which results in the workload identity injected SA token to not be used for URL signing due to which a lot of velero features are not working anymore including snapshot backups, logs, describing, … Removing the related code from the PR and running a custom image fixes the issue as currently the Workload Identity SA is detected as external_account and URL signing is not even attempted.

Error:

time="2023-10-05T09:37:42Z" level=warning msg="fail to get Backup metadata file's download URL {BackupLog test4}, retry later: rpc error: code = Unknown desc = cannot sign blob using non SA file credentials" controller=download-request downloadRequest=velero-dev/test4-cb2a5ec8-9a29-4b5d-9de2-2cea227abcc0 logSource="pkg/controller/download_request_controller.go:206"

What did you expect to happen:

I expect to be able to use Workload Identities as I was able to in 1.11.1 and also in 1.12.0 with modifications.

The following information will help us better understand what’s going on:

If you are using velero v1.7.0+:
Don’t expect too much except timeouts as the URL can’t be signed since that step is skipped. bundle-2023-10-05-13-27-29.tar.gz

Anything else you would like to add:

Looks to me like a regression in the plugin and wasn’t ever tested against workload identities.

Environment:

  • Velero version (use velero version): 1.12.0
  • Velero features (use velero client config get features): NOT SET
  • Kubernetes version (use kubectl version): v1.27.3-gke.1700 (server) - v1.28.2 (client)
  • Kubernetes installer & version: v1.27.3-gke.1700
  • Cloud provider or hardware configuration: GCP (GKE)
  • OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project’s top voted issues listed here.
Use the “reaction smiley face” up to the right of this comment to vote.

  • 👍 for “I would like to see this bug fixed as soon as possible”
  • 👎 for “There are more important bugs to focus on right now”

About this issue

  • Original URL
  • State: closed
  • Created 9 months ago
  • Reactions: 4
  • Comments: 22 (13 by maintainers)

Most upvoted comments

Updating docs

but not this whole implicit access. (at least didn’t find a function yet in the API docs)

got it.

Why do we have to support identity federation anyway

My team needed it for our customers 😃

I have to say that AWS equivalent called IAM Roles for Service Accounts is much easier to detect … never thought I would ever say that. For GKE, there is not really a good indicator … just checked the mutated pod and there’s no way to figure it out. Maybe there’s a function? Opposed to AWS they’re not using any envs to configure it.

The SDK is making use of this --> https://cloud.google.com/docs/authentication/application-default-credentials