argo-cd: Unable to use OCI Registry with sub helm charts

Hi, I encounter very similar error that is described in origin of this thread and also in thread #11717.

After upgrade from ArgoCD v2.7.7 to v2.7.8 (and any upper version, my main destination is latest v2.8.2) I see this error

(v2.8.0+)

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://docker.bcas.cz/cz/bcas/app/crm/crm/helm/crm: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://gitlab.bcas.cz/jwt/auth?scope=repository%3Acz%2Fbcas%2Fapp%2Fcrm%2Fcrm%2Fhelm%2Fcrm%3Apull&service=container_registry: 403 Forbidden

(v2.7.8-2.7.X)

rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://docker.bcas.cz/cz/bcas/app/broker-coach/broker-coach/helm/broker-coach: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

at my Applications which depend on some Helm Chart in OCI repository. When I create directly Application from OCI repo, then Argo will download and build Helm Chart correctly. But when it is depended in some parent Chart, then Argo cannot download it during build and throw this error.

Repository seems to be OK.

Project hase assigned this repository as source repository.

This is what my helm dependenci definition looks like

And this is what my repository secret for Argo looks like

apiVersion: v1
kind: Secret
metadata:
  annotations:
  labels:
    argocd.argoproj.io/secret-type: repository
  name: argocd-repo-cz.bcas-helm
  namespace: argo-cd
type: Opaque
data:
  enableOCI: enable
  name: cz.bcas-helm
  password: XXXXXXXXXXXXXX
  project: applications
  type: helm
  url: docker.bcas.cz/cz/bcas
  username: XXXXXXXXXXXXXX

Is there anyone still working on this problem? And can it help with debugging and fixing?

I’ve been digging through all of the related issues that I can find on this, and I’m in the same position as others.

Our scenario:

1. We have helm chart dependencies in Google Artifact Registry using OCI
2. In kubernetes (GKE), we rely on workload identities to grant rights to individual pods / workloads
3. In ArgoCD, we apply a helm chart from git which includes the OCI dependency chart
4. helm dependency build fails with a 403 due to an anonymous token being passed

Expectation:

Since the pod(s) running ArgoCD applications have workload identities granted via IAM, we should not need to pass credentials explicitly. The fact that the pod has proper bindings to IAM means that the request should essentially be possible by using the access token method where the token is granted by the GCP metadata server upon request.

Reference: https://cloud.google.com/artifact-registry/docs/helm/authentication

Related: https://github.com/argoproj/argo-cd/pull/12554 https://github.com/argoproj/argo-cd/issues/12392

I’m currently facing exactly the same problem as described by @jascsch We’re also forced to use OCI but when I try to use it as dependency, I’m also always getting a forbidden message, although it’s public and I can fetch it using helm pull.

So I also think that this should be an issue of argo. Fixing this issue is very appreciated and import for me, too.

We have a workaround for this issue. Instead of using a Chart.yaml file we decided to specify multiple sources and reference the helm chart as a git repo like this:

kind: Application
metadata:
  name: msag-clamav-dev
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: 'https://git.i.mercedes-benz.com/LES/msag-clamav'
      targetRevision: develop
      ref: myRepo
      helm:
        releaseName: msag-clamav
    - repoURL: 'https://git.i.mercedes-benz.com/LES/helm.git'
      path: charts/les-service
      targetRevision: main
      helm:
        releaseName: msag-clamav
        valueFiles:
          - $myRepo/env/values-dev.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: dev

I have a workaround for this issue that applies to GCP / Google Artifact Registry when using workload identities:

1. Build a new container with gcloud: Dockerfile

FROM google/cloud-sdk:latest as gcloudSDK

FROM argoproj/argocd:v2.6.0

COPY --from=gcloudSDK /usr/lib/google-cloud-sdk /usr/lib/google-cloud-sdk

# switch to root
USER 0

RUN \
  ln -s /usr/lib/google-cloud-sdk/bin/gcloud /usr/bin/gcloud && \
  ln -s /usr/lib/google-cloud-sdk/bin/docker-credential-gcloud /usr/bin/docker-credential-gcloud

RUN \
  apt-get update && apt-get install -f -y curl

# switch back to argocd
USER 999

1. Modify argo-cd-repo-server (example given for helm) values.yaml

  repoServer:
    # Default image used by all components
    image:
      # -- If defined, a repository applied to all Argo CD deployments
      repository: us-docker.pkg.dev/myregisry/container/argocd
      # -- Overrides the global Argo CD image tag whose default is the chart appVersion
      tag: v2.6.0-gcloud
      # -- If defined, a imagePullPolicy applied to all Argo CD deployments
      imagePullPolicy: IfNotPresent

    serviceAccount:
      annotations:
        iam.gke.io/gcp-service-account: argo@myproject.iam.gserviceaccount.com

    securityContext:
      runAsUser: 999

    volumes:
      - name: gcloud
        emptyDir: {}

    volumeMounts:
      - mountPath: /gcloud
        name: gcloud

    env:
      - name: CLOUDSDK_CONFIG
        value: /gcloud/
      - name: DOCKER_CONFIG
        value: /helm-working-dir/.docker

    initContainers:
      - name: activate-helm-creds
        image: us-docker.pkg.dev/myregisry/container/argocd:v2.6.0-gcloud

        securityContext:
          runAsUser: 0

        volumeMounts:
          - mountPath: /helm-working-dir
            name: helm-working-dir
          - mountPath: /gcloud
            name: gcloud

        env:
          - name: CLOUDSDK_CONFIG
            value: /gcloud/
          - name: DOCKER_CONFIG
            value: /helm-working-dir/.docker

        command: ["/bin/bash", "-c"]
        args:
          - gcloud init;
            ( yes | gcloud auth configure-docker || test $? -eq 141 );
            ( yes | gcloud auth configure-docker us-docker.pkg.dev || test $? -eq 141 );
            chown -R 999:999 /gcloud;
            chown -R 999:999 /helm-working-dir;
            find /gcloud -type f -exec chmod +r {} \; ;
            find /gcloud -type d -exec chmod +rx {} \; ;
            find /helm-working-dir -type f -exec chmod +r {} \; ;
            find /helm-working-dir -type d -exec chmod +rx {} \; ;

1. Setup workload identity bindings (example given for ConfigConnector)

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPartialPolicy
metadata:
  name: sa-argo-cnrm
spec:
  resourceRef:
    apiVersion: iam.cnrm.cloud.google.com/v1beta1
    kind: IAMServiceAccount
    external: projects/myproject/serviceAccounts/argocd@myproject.iam.gserviceaccount.com
  bindings:
    - role: roles/iam.workloadIdentityUser
      members:
        - member: serviceAccount:myproject.svc.id.goog[argocd/argo-cd-repo-server]

1. Disable helm.passCredentials for your application managing deployments
2. Sync your failing application. If an error is cached, you may need to delete / recreate the offending app.