argo-cd: OCI helm charts hosted in AWS OCI broken after upgrade from v2.0.4 to v2.2.2

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

[ x] I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[ x] I’ve included steps to reproduce the bug.
[x ] I’ve pasted the output of argocd version.

Describe the bug Application Chart:

apiVersion: v3
name: react-frontend-service
version: 0.2.7-latest-a6f42056
dependencies:
  - name: react-frontend-service
    repository: oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service
    version: 0.2.7-latest-a6f42056

$ argocd --server argocd-test.falkor.rocks --grpc-web repo list
TYPE  NAME           REPO                                                              INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
helm  us-west-2-oci  712053168757.dkr.ecr.us-west-2.amazonaws.com                      false     true   false  true   Successful

All applications referencing AWS oci charts fail after upgrade. It appears “helm registry login” is only called when add the helm repo using “helm repo add” from cronjob. The “helm dependency update” fails and no login messages appear in repoServer logs.

Error message from UI:

rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service/react-frontend-service: pulling from host 712053168757.dkr.ecr.us-west-2.amazonaws.com failed with status code [manifests 0.2.7-latest-a6f42056]: 401 Unauthorized

Repo server log:

time="2022-01-18T22:25:22Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service/react-frontend-service: pulling from host 712053168757.dkr.ecr.us-west-2.amazonaws.com failed with status code [manifests 0.2.7-latest-a6f42056]: 401 Unauthorized" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-01-18T22:30:22Z" grpc.service=repository.RepoServerService grpc.start_time="2022-01-18T22:25:22Z" grpc.time_ms=2.77 span.kind=server system=grpc

If I bash into the repo server and do “helm registry login” and then “help dependency build” in the /tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base folder, it successfully pulls the OCI helm chart. So no permission issues. I’m using username/password/url from the argocd cli helm repo add created kubernetes secret repo-4159968007.

This also happens if I upgrade to 2.1.7 instead of 2.2.2. Post in argocd slack also: slack

To Reproduce

Add Application using chart with reference to AWS OCI helm chart.

Expected behavior

Helm charts should pull from AWS OCI after successful “helm registry login” executed before running “helm dependency update”.

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v2.2.2+03b17e0.dirty
  BuildDate: 2022-01-01T16:53:02Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: dirty
  GoVersion: go1.17.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.2.2+03b17e0
  BuildDate: 2022-01-01T06:27:52Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.22.2
  Jsonnet Version: v0.17.0

Logs

time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=1232.725993
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git checkout --force 4bf8341c9c979b45c443870bfff1328a49536dd1]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=4.994643
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git clean -fdx]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=3.643158
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git rev-parse HEAD]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=2.335853
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[helm template . --name-template react-frontend-service-a0000000027 --namespace example-test --kube-version 1.21 --values ../overlays/a0000000027/globals.yaml --values ../overlays/flags.yaml --values ../overlays/a0000000027/values.yaml --values ../overlays/a0000000027/hotfix.yaml --api-versions acme.cert-manager.io/v1 --api-versions acme.cert-manager.io/v1/Challenge --api-versions acme.cert-manager.io/v1/Order --api-versions acme.cert-manager.io/v1alpha2 --api-versions acme.cert-manager.io/v1alpha2/Challenge --api-versions acme.cert-manager.io/v1alpha2/Order --api-versions acme.cert-manager.io/v1alpha3 --api-versions acme.cert-manager.io/v1alpha3/Challenge --api-versions acme.cert-manager.io/v1alpha3/Order --api-versions acme.cert-manager.io/v1beta1 --api-versions acme.cert-manager.io/v1beta1/Challenge --api-versions acme.cert-manager.io/v1beta1/Order --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1beta1 --api-versions admissionregistration.k8s.io/v1beta1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1beta1/ValidatingWebhookConfiguration --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1/CustomResourceDefinition --api-versions apiextensions.k8s.io/v1beta1 --api-versions apiextensions.k8s.io/v1beta1/CustomResourceDefinition --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1/APIService --api-versions apiregistration.k8s.io/v1beta1 --api-versions apiregistration.k8s.io/v1beta1/APIService --api-versions apps/v1 --api-versions apps/v1/ControllerRevision --api-versions apps/v1/DaemonSet --api-versions apps/v1/Deployment --api-versions apps/v1/ReplicaSet --api-versions apps/v1/StatefulSet --api-versions argoproj.io/v1alpha1 --api-versions argoproj.io/v1alpha1/AppProject --api-versions argoproj.io/v1alpha1/Application --api-versions argoproj.io/v1alpha1/ApplicationSet --api-versions autoscaling/v1 --api-versions autoscaling/v1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta1 --api-versions autoscaling/v2beta1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta2 --api-versions autoscaling/v2beta2/HorizontalPodAutoscaler --api-versions batch/v1 --api-versions batch/v1/CronJob --api-versions batch/v1/Job --api-versions batch/v1beta1 --api-versions batch/v1beta1/CronJob --api-versions bitnami.com/v1alpha1 --api-versions bitnami.com/v1alpha1/SealedSecret --api-versions cert-manager.io/v1 --api-versions cert-manager.io/v1/Certificate --api-versions cert-manager.io/v1/CertificateRequest --api-versions cert-manager.io/v1/ClusterIssuer --api-versions cert-manager.io/v1/Issuer --api-versions cert-manager.io/v1alpha2 --api-versions cert-manager.io/v1alpha2/Certificate --api-versions cert-manager.io/v1alpha2/CertificateRequest --api-versions cert-manager.io/v1alpha2/ClusterIssuer --api-versions cert-manager.io/v1alpha2/Issuer --api-versions cert-manager.io/v1alpha3 --api-versions cert-manager.io/v1alpha3/Certificate --api-versions cert-manager.io/v1alpha3/CertificateRequest --api-versions cert-manager.io/v1alpha3/ClusterIssuer --api-versions cert-manager.io/v1alpha3/Issuer --api-versions cert-manager.io/v1beta1 --api-versions cert-manager.io/v1beta1/Certificate --api-versions cert-manager.io/v1beta1/CertificateRequest --api-versions cert-manager.io/v1beta1/ClusterIssuer --api-versions cert-manager.io/v1beta1/Issuer --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1/CertificateSigningRequest --api-versions certificates.k8s.io/v1beta1 --api-versions certificates.k8s.io/v1beta1/CertificateSigningRequest --api-versions config.gatekeeper.sh/v1alpha1 --api-versions config.gatekeeper.sh/v1alpha1/Config --api-versions constraints.gatekeeper.sh/v1alpha1 --api-versions constraints.gatekeeper.sh/v1alpha1/K8sContainerLimits --api-versions constraints.gatekeeper.sh/v1alpha1/K8sNodeSelectors --api-versions constraints.gatekeeper.sh/v1alpha1/K8sPSPPrivilegedContainer --api-versions constraints.gatekeeper.sh/v1alpha1/K8sPSPVolumeTypes --api-versions constraints.gatekeeper.sh/v1alpha1/K8sRequiredLabels --api-versions constraints.gatekeeper.sh/v1alpha1/K8sRequiredProbes --api-versions constraints.gatekeeper.sh/v1alpha1/K8sUniqueIngressHost --api-versions constraints.gatekeeper.sh/v1alpha1/K8sUniqueServiceSelector --api-versions constraints.gatekeeper.sh/v1beta1 --api-versions constraints.gatekeeper.sh/v1beta1/K8sContainerLimits --api-versions constraints.gatekeeper.sh/v1beta1/K8sNodeSelectors --api-versions constraints.gatekeeper.sh/v1beta1/K8sPSPPrivilegedContainer --api-versions constraints.gatekeeper.sh/v1beta1/K8sPSPVolumeTypes --api-versions constraints.gatekeeper.sh/v1beta1/K8sRequiredLabels --api-versions constraints.gatekeeper.sh/v1beta1/K8sRequiredProbes --api-versions constraints.gatekeeper.sh/v1beta1/K8sUniqueIngressHost --api-versions constraints.gatekeeper.sh/v1beta1/K8sUniqueServiceSelector --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1/Lease --api-versions coordination.k8s.io/v1beta1 --api-versions coordination.k8s.io/v1beta1/Lease --api-versions crd.k8s.amazonaws.com/v1alpha1 --api-versions crd.k8s.amazonaws.com/v1alpha1/ENIConfig --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1/EndpointSlice --api-versions discovery.k8s.io/v1beta1 --api-versions discovery.k8s.io/v1beta1/EndpointSlice --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1/Event --api-versions events.k8s.io/v1beta1 --api-versions events.k8s.io/v1beta1/Event --api-versions extensions/v1beta1 --api-versions extensions/v1beta1/Ingress --api-versions flowcontrol.apiserver.k8s.io/v1beta1 --api-versions flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration --api-versions koudingspawn.de/v1 --api-versions koudingspawn.de/v1/Vault --api-versions linkerd.io/v1alpha1 --api-versions linkerd.io/v1alpha1/ServiceProfile --api-versions linkerd.io/v1alpha2 --api-versions linkerd.io/v1alpha2/ServiceProfile --api-versions mutations.gatekeeper.sh/v1alpha1 --api-versions mutations.gatekeeper.sh/v1alpha1/Assign --api-versions mutations.gatekeeper.sh/v1alpha1/AssignMetadata --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1/Ingress --api-versions networking.k8s.io/v1/IngressClass --api-versions networking.k8s.io/v1/NetworkPolicy --api-versions networking.k8s.io/v1beta1 --api-versions networking.k8s.io/v1beta1/Ingress --api-versions networking.k8s.io/v1beta1/IngressClass --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1/RuntimeClass --api-versions node.k8s.io/v1beta1 --api-versions node.k8s.io/v1beta1/RuntimeClass --api-versions policy.linkerd.io/v1alpha1 --api-versions policy.linkerd.io/v1alpha1/Server --api-versions policy.linkerd.io/v1alpha1/ServerAuthorization --api-versions policy.linkerd.io/v1beta1 --api-versions policy.linkerd.io/v1beta1/Server --api-versions policy.linkerd.io/v1beta1/ServerAuthorization --api-versions policy/v1 --api-versions policy/v1/PodDisruptionBudget --api-versions policy/v1beta1 --api-versions policy/v1beta1/PodDisruptionBudget --api-versions policy/v1beta1/PodSecurityPolicy --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1/ClusterRole --api-versions rbac.authorization.k8s.io/v1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1/Role --api-versions rbac.authorization.k8s.io/v1/RoleBinding --api-versions rbac.authorization.k8s.io/v1beta1 --api-versions rbac.authorization.k8s.io/v1beta1/ClusterRole --api-versions rbac.authorization.k8s.io/v1beta1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1beta1/Role --api-versions rbac.authorization.k8s.io/v1beta1/RoleBinding --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1/PriorityClass --api-versions scheduling.k8s.io/v1beta1 --api-versions scheduling.k8s.io/v1beta1/PriorityClass --api-versions split.smi-spec.io/v1alpha1 --api-versions split.smi-spec.io/v1alpha1/TrafficSplit --api-versions split.smi-spec.io/v1alpha2 --api-versions split.smi-spec.io/v1alpha2/TrafficSplit --api-versions status.gatekeeper.sh/v1beta1 --api-versions status.gatekeeper.sh/v1beta1/ConstraintPodStatus --api-versions status.gatekeeper.sh/v1beta1/ConstraintTemplatePodStatus --api-versions status.gatekeeper.sh/v1beta1/MutatorPodStatus --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1/CSIDriver --api-versions storage.k8s.io/v1/CSINode --api-versions storage.k8s.io/v1/StorageClass --api-versions storage.k8s.io/v1/VolumeAttachment --api-versions storage.k8s.io/v1beta1 --api-versions storage.k8s.io/v1beta1/CSIDriver --api-versions storage.k8s.io/v1beta1/CSINode --api-versions storage.k8s.io/v1beta1/CSIStorageCapacity --api-versions storage.k8s.io/v1beta1/StorageClass --api-versions storage.k8s.io/v1beta1/VolumeAttachment --api-versions templates.gatekeeper.sh/v1 --api-versions templates.gatekeeper.sh/v1/ConstraintTemplate --api-versions templates.gatekeeper.sh/v1alpha1 --api-versions templates.gatekeeper.sh/v1alpha1/ConstraintTemplate --api-versions templates.gatekeeper.sh/v1beta1 --api-versions templates.gatekeeper.sh/v1beta1/ConstraintTemplate --api-versions v1 --api-versions v1/ConfigMap --api-versions v1/Endpoints --api-versions v1/Event --api-versions v1/LimitRange --api-versions v1/Namespace --api-versions v1/Node --api-versions v1/PersistentVolume --api-versions v1/PersistentVolumeClaim --api-versions v1/Pod --api-versions v1/PodTemplate --api-versions v1/ReplicationController --api-versions v1/ResourceQuota --api-versions v1/Secret --api-versions v1/Service --api-versions v1/ServiceAccount --api-versions vpcresources.k8s.aws/v1beta1 --api-versions vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy --include-crds]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base operation_name="exec helm" time_ms=38.575524
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[helm dependency build]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base operation_name="exec helm" time_ms=96.118729

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 15 (3 by maintainers)

Most upvoted comments

Have a workaround using plugin. In values.yaml:

  repoServer:
    volumes:
      - name: custom-tools
        emptyDir: {}
      - name: plugin-scripts
        configMap:
          name: plugin-scripts
          defaultMode: 0755
    initContainers:
    - name: helm-setup
      image: alpine:3
      command: [
        "/bin/sh",
        "-c",
        "apk add --no-cache aws-cli;
        which aws;
        aws --version;
        cp /usr/bin/aws /custom-tools/aws; "]
      volumeMounts:
      - mountPath: /custom-tools
        name: custom-tools
    volumeMounts:
    - mountPath: /usr/local/bin/aws
      name: custom-tools
      subPath: aws
    - mountPath: /usr/local/bin/helmsetup.sh
      name: plugin-scripts
      subPath: helmsetup.sh
      readOnly: true
    - mountPath: /usr/local/bin/helmtemplate.sh
      name: plugin-scripts
      subPath: helmtemplate.sh
      readOnly: true

  server:
    config:
      configManagementPlugins: |
        - name: helm_aws_oci
          init:
            command: ["/bin/sh", "-c", "/usr/local/bin/helmsetup.sh"]
          generate:
            command: ["/bin/sh", "-c", "/usr/local/bin/helmtemplate.sh"]

Add scripts configmap under templates/plugin-scripts.yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: plugin-scripts
data:
  helmsetup.sh: |
    #!/bin/sh
    export HELM_EXPERIMENTAL_OCI=1
    aws ecr get-login-password --region us-west-2 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-west-2.amazonaws.com"
    helm dependency build
  helmtemplate.sh: |
    #!/bin/sh
    export HELM_EXPERIMENTAL_OCI=1
    helm template --api-versions ${KUBE_API_VERSIONS} -n ${ARGOCD_APP_NAMESPACE} ${ARGOCD_APP_NAME} . --include-crds ${HELM_VALUES_FILES} ${HELM_ARGS}

In application def, replace helm section with plugin:

apiVersion: argoproj.io/v1alpha1
kind: Application
*******
      source:
        path: ./lacework/helm_base
        plugin:
          env:
          - name: HELM_VALUES_FILES
            value: --values ../overlays/a0000000027/globals.yaml --values ../overlays/flags.yaml
              --values ../overlays/a0000000027/values.yaml --values ../overlays/a0000000027/hotfix.yaml
          - name: HELM_ARGS
            value: --set lacework.lacework-agent.laceworkConfig.accessToken='**************************'
              --set lacework.lacework-agent.laceworkConfig.env='mushu.falkor.rocks'
              --set lacework.lacework-agent.laceworkConfig.kubernetesCluster='a0000000027'
          name: helm_aws_oci

pthornton on Feb 10, 2022