argo-workflows: `401 Unauthorized` when looking for entrypoint/cmd of an image hosted on a private AWS ECR with v3.6.0

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I’d like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

I’m upgrading from v3.5.12 to v3.6.0 using the official Helm chart. After the upgrade, workflows with images hosted on ECR fail to start.

I always have this error:

failed to look-up entrypoint/cmd for image "<account-id>.dkr.ecr.<region>.amazonaws.com/<image>:<version>", 
you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: 
GET https://<account-id>.dkr.ecr.<region>.amazonaws.com/v2/<image>/manifests/<version>: unexpected status code 401 Unauthorized: Not Authorized

I precise that my image does contains an ENTRYPOINT. I tried to give IAM permissions to read the ECR repository to the workflow-controller with IRSA but it doesn’t work.

Version(s)

v3.6.0

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

Any simple workflow whose image is hosted on a private ECR repository.

Logs from the workflow controller

{"Phase":"","ResourceVersion":"714280881","level":"info","msg":"Processing workflow","namespace":"simple-workflow","time":"2024-11-27T06:56:44.261Z","workflow":"mycli-l5rc4"}
{"base":"*v1alpha1.Workflow (namespace=simple-workflow,name=mycli-l5rc4)","level":"debug","msg":"Resolving the template","time":"2024-11-27T06:56:44.268Z","tmpl":"*v1alpha1.WorkflowStep (mycli/mycli#false)"}
{"base":"*v1alpha1.Workflow (namespace=simple-workflow,name=mycli-l5rc4)","level":"debug","msg":"Found stored template","time":"2024-11-27T06:56:44.268Z","tmpl":"*v1alpha1.WorkflowStep (mycli/mycli#false)"}
{"base":"*v1alpha1.Workflow (namespace=simple-workflow,name=mycli-l5rc4)","level":"debug","msg":"Resolving the template","time":"2024-11-27T06:56:44.274Z","tmpl":"*v1alpha1.WorkflowStep (mycli/mycli#false)"}
{"base":"*v1alpha1.Workflow (namespace=simple-workflow,name=mycli-l5rc4)","level":"debug","msg":"Found stored template","time":"2024-11-27T06:56:44.274Z","tmpl":"*v1alpha1.WorkflowStep (mycli/mycli#false)"}
{"level":"info","msg":"Task-result reconciliation","namespace":"simple-workflow","numObjs":0,"time":"2024-11-27T06:56:44.297Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"Skipping artifact GC","namespace":"simple-workflow","time":"2024-11-27T06:56:44.297Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"Updated phase  -\u003e Running","namespace":"simple-workflow","time":"2024-11-27T06:56:44.297Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"simple-workflow\", Name:\"mycli-l5rc4\", UID:\"8f9e51d3-2334-40f9-93f0-2f2f3321903f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"714280881\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running","time":"2024-11-27T06:56:44.298Z"}
{"level":"debug","msg":"Evaluating node mycli-l5rc4: template: *v1alpha1.WorkflowStep (mycli), boundaryID: ","namespace":"simple-workflow","time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"level":"warning","msg":"Node was nil, will be initialized as type Skipped","namespace":"simple-workflow","time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"Initializing node mycli-l5rc4: template: *v1alpha1.WorkflowStep (mycli), boundaryID: ","namespace":"simple-workflow","time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"was unable to obtain node for , letting display name to be nodeName","namespace":"simple-workflow","time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"Pod node mycli-l5rc4 initialized Pending","namespace":"simple-workflow","time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"Executing node mycli-l5rc4 with container template: mycli\n","namespace":"simple-workflow","time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"","namespace":"simple-workflow","needLocation":false,"time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"level":"warning","msg":"couldn't get boundaryTemplate through nodeName mycli-l5rc4","namespace":"simple-workflow","time":"2024-11-27T06:56:44.303Z","workflow":"mycli-l5rc4"}
{"error":"failed to look-up entrypoint/cmd for image \"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\n","level":"error","msg":"Mark error node","namespace":"simple-workflow","nodeName":"mycli-l5rc4","time":"2024-11-27T06:56:44.333Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"node mycli-l5rc4 phase Pending -\u003e Error","namespace":"simple-workflow","time":"2024-11-27T06:56:44.333Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"node mycli-l5rc4 message: failed to look-up entrypoint/cmd for image \"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\n","namespace":"simple-workflow","time":"2024-11-27T06:56:44.333Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"node mycli-l5rc4 finished: 2024-11-27 06:56:44.333377165 +0000 UTC","namespace":"simple-workflow","time":"2024-11-27T06:56:44.333Z","workflow":"mycli-l5rc4"}
{"error":"failed to look-up entrypoint/cmd for image \"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\n","level":"error","msg":"error in entry template execution","namespace":"simple-workflow","time":"2024-11-27T06:56:44.333Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"Task results completion status: map[]","namespace":"simple-workflow","time":"2024-11-27T06:56:44.333Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"Updated phase Running -\u003e Error","namespace":"simple-workflow","time":"2024-11-27T06:56:44.333Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"Updated message  -\u003e error in entry template execution: failed to look-up entrypoint/cmd for image \"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\n","namespace":"simple-workflow","time":"2024-11-27T06:56:44.334Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"Marking workflow completed","namespace":"simple-workflow","time":"2024-11-27T06:56:44.334Z","workflow":"mycli-l5rc4"}
{"level":"info","msg":"Marking workflow as pending archiving","namespace":"simple-workflow","time":"2024-11-27T06:56:44.334Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"simple-workflow\", Name:\"mycli-l5rc4\", UID:\"8f9e51d3-2334-40f9-93f0-2f2f3321903f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"714280881\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowFailed' error in entry template execution: failed to look-up entrypoint/cmd for image \"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\n","time":"2024-11-27T06:56:44.334Z"}
{"level":"debug","msg":"Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"workflows.argoproj.io/archive-strategy\":\"false\",\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Error\",\"workflows.argoproj.io/workflow-archiving-status\":\"Pending\"}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{\"s3\":{\"bucket\":\"my-argo-workflows-artifacts\",\"endpoint\":\"s3.amazonaws.com\",\"insecure\":false,\"keyFormat\":\"{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{workflow.name}}/{{pod.name}}\",\"region\":\"eu-west-3\",\"useSDKCreds\":true}},\"default\":true},\"conditions\":[{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2024-11-27T06:56:44Z\",\"message\":\"error in entry template execution: failed to look-up entrypoint/cmd for image \\\"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\\\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\\n\",\"nodes\":{\"mycli-l5rc4\":{\"displayName\":\"mycli-l5rc4\",\"finishedAt\":\"2024-11-27T06:56:44Z\",\"id\":\"mycli-l5rc4\",\"message\":\"failed to look-up entrypoint/cmd for image \\\"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\\\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\\n\",\"name\":\"mycli-l5rc4\",\"phase\":\"Error\",\"startedAt\":\"2024-11-27T06:56:44Z\",\"templateName\":\"mycli\",\"templateScope\":\"local/\",\"type\":\"Pod\"}},\"phase\":\"Error\",\"startedAt\":\"2024-11-27T06:56:44Z\",\"storedWorkflowTemplateSpec\":{\"activeDeadlineSeconds\":21600,\"arguments\":{},\"entrypoint\":\"mycli\",\"podGC\":{\"deleteDelayDuration\":\"24h\",\"strategy\":\"OnWorkflowCompletion\"},\"templates\":[{\"container\":{\"args\":[\"--version\"],\"image\":\"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\",\"name\":\"\",\"resources\":{}},\"inputs\":{},\"metadata\":{},\"name\":\"mycli\",\"outputs\":{},\"serviceAccountName\":\"simple-workflow\"}],\"ttlStrategy\":{\"secondsAfterCompletion\":259200},\"workflowMetadata\":{\"labels\":{\"workflows.argoproj.io/archive-strategy\":\"false\"}},\"workflowTemplateRef\":{\"name\":\"mycli\"}}}}","time":"2024-11-27T06:56:44.334Z"}
{"level":"info","msg":"Workflow update successful","namespace":"simple-workflow","phase":"Error","resourceVersion":"714280885","time":"2024-11-27T06:56:44.346Z","workflow":"mycli-l5rc4"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"simple-workflow\", Name:\"mycli-l5rc4\", UID:\"8f9e51d3-2334-40f9-93f0-2f2f3321903f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"714280885\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node mycli-l5rc4: failed to look-up entrypoint/cmd for image \"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\n","time":"2024-11-27T06:56:44.346Z"}
{"level":"debug","msg":"Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"simple-workflow\", Name:\"mycli-l5rc4\", UID:\"8f9e51d3-2334-40f9-93f0-2f2f3321903f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"714280885\", FieldPath:\"\"}): type: 'Warning' reason: 'WorkflowNodeError' Error node mycli-l5rc4: failed to look-up entrypoint/cmd for image \"012345678910.dkr.ecr.eu-west-3.amazonaws.com/mycli:2.1.0\", you must either explicitly specify the command, or list the image's command in the index: https://argo-workflows.readthedocs.io/en/latest/workflow-executors/#emissary-emissary: GET https://012345678910.dkr.ecr.eu-west-3.amazonaws.com/v2/mycli/manifests/2.1.0: unexpected status code 401 Unauthorized: Not Authorized\n","time":"2024-11-27T06:56:44.346Z"}
{"level":"info","msg":"archiving workflow","namespace":"simple-workflow","time":"2024-11-27T06:56:44.354Z","uid":"8f9e51d3-2334-40f9-93f0-2f2f3321903f","workflow":"mycli-l5rc4"}
{"level":"info","msg":"Queueing Error workflow simple-workflow/mycli-l5rc4 for delete in 72h0m0s due to TTL","time":"2024-11-27T06:56:44.412Z"}

Logs from in your workflow’s wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 19
  • Comments: 33 (21 by maintainers)

Most upvoted comments

When could we expect this fix to be released? Is there any blocker? This seems to be a big issue for anyone using private ecr registries.

https://github.com/google/go-containerregistry/releases/tag/v0.20.3 is released and fixes this issue. If we upgrade our dependencies, it should fix it.

I just tested 3.6.5 and it seems the issue was fixed

3.6.5 is now released, so let me know if this is fixed please.

Hi here is a detailed summary explaining why and how we fixed our error “401 Unauthorized when looking for entrypoint/cmd of an image hosted on a private AWS ECR” after upgrading the infrastructure and configuration of our eks cluster. Before the upgrade argo workflow (v3.6.5) accessing a private AWS ECR registry worked fine without any issues.

Software Upgrade

with the new availability of EKS 1.33, we decided to also migrate our eks nodes from Amazon Linux 2 to Amazon Linux 2023, so our upgrade included some major changes

  • EKS 1.30 → 1.33
  • Amazon Linux 2 → Amazon Linux 2023
  • IRSA → AWS EKS Pod Identity
  • Argo Workflows v3.6.5 → v3.6.7

After the upgrade we „surprisingly“ encountered the error when running workflows using images stored in our private ECR: "…. unexpected status code 401 Unauthorized:“ and I found this github issue. Except argo workflow everything worked perfectly after the upgrade including k8s Job/Cronjob and it took me quite a while to figure out what happened in the background after the upgrade and I also was misleaded by the comments in this thread.

Root Cause:

  • Amazon Linux 2023 disables instance metadata service v1 (IMDSv1) by default and we didn’t increase the hop limit
  • previously, our Amazon Linux 2 eks nodes loaded configured iam instance profiles with ECR read permissions via IMDSv1.
  • with the switch to IMDSv2-only and pod identity replacing IRSA, the workflow pods (and controller!!!) no longer had credentials to access our ECR.

Resolution

to fix error 401 you just need to configure explicit iam roles for both the workflow controller and the workflow pods with proper ECR permissions:

Step-by-Step fix using Pod Identity

1. Create an iam role for ECR readonly access

aws iam create-role \
   --role-name PodIdentityECRReadOnlyRole \
   --assume-role-policy-document file://trust-policy.json

The trust-policy.json defines the trust relationship for eks pod identity.

2. Attach AWS managed read-only ECR policy

aws iam attach-role-policy \
   --role-name PodIdentityECRReadOnlyRole \
   --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

This gives the full readonly role permissions to pull images from our private ECR repositories.

3. Associate the role with the argo workflow controller service account

aws eks create-pod-identity-association \
   --cluster-name SandboxEks \
   --namespace argo \
   --service-account argo \
   --role-arn arn:aws:iam::<account-id>:role/PodIdentityECRReadOnlyRole

Important: the argo workflow controller itself needs ECR read permissions to fetch image metadata (entrypoint/cmd) when using the emissary executor!

4. Associate a role for workflow pods (cluster workflow template) with its service account

aws eks create-pod-identity-association \
  --cluster-name SandboxEks \
  --namespace MarketData \
  --service-account argo-cluster-workflow-sa \
  --role-arn arn:aws:iam::<account-id>:role/GenericPodCustomAccess

This allows the actual workflow pods to assume the correct iam role, in this case GenericPodCustomAccess and access any needed AWS services or secrets during job execution. Once the controller and workflow pods had the correct iam roles via pod identity, including ECR read-only permissions, the 401 Unauthorized errors disappeared and argo workflows could again successfully fetch metadata and run workloads using private ECR images.

Key takeaway

Regardless of whether you use AWS EKS Pod Identity, IRSA or IMDS instance profiles, the argo Workflow Controller must have read access to ECR when using the Emissary Executor, as it checks the metadata of the image (e.g. the entry point) by querying ECR at the time of submission - not just during pod execution. Just make sure the argo controller really has the permission.

I hope these hints are helpful, if anyone has the same problem with AWS and a private ECR A R

Also tested v3.6.5 and the issue is fixed. thank you 🙏🏾

@Joibel Did not work for me

This should be fixed in 3.6.5, I will release it soon.

A related comment: https://github.com/argoproj/argo-workflows/pull/14008#issuecomment-2650115475

I failed to backport this (it fails to automatically backport and I forgot the manual version). It’s in progress and will be in the next 3.6 release, which I’ll try to do sooner rather than later.

Still happening in 3.6.4