argo-workflows: Emissary does not recognize the Entrypoint in the Docker image when unverified cert
Hello,
We have recently upgraded to Argo Workflows 3.3.8 and changed the executor to Emissary. We are using Argo since about two years and built a framework around it. Until now we have used pns executor. After switching to Emissary, all our workflows fail with:
task <task name> errored: container "main" in template <template>, does not have the command specified: when using the emissary executor you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary
We always provide ENTRYPOINT and CMD in the Dockerfile and override the CMD with args in the container spec in the workflow template.
I have seen in this thread (https://github.com/argoproj/argo-workflows/pull/8345#issuecomment-1180223024) that you expect it to work this way:
- If you do not supply command or args for a Container, the defaults defined in the Docker image are used.
- If you supply only args for a Container, the default Entrypoint defined in the Docker image is run with the args that you supplied.
- If you supply a command for a Container, only the supplied command is used. The default EntryPoint and the default Cmd defined in the Docker image are ignored. Your command is run with the args supplied (or no args if none supplied).
However the second point (ENTRYPOINT in the Dockerfile and no command in the Container) does not work. Probably you did not have a test case for this one, I am attaching my simple test case for you then.
Could you please confirm this is the bug which must be fixed?
_Originally posted by @vitalyrychkov in https://github.com/argoproj/argo-workflows/issues/8345#issuecomment-1180223024_
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 29
- Comments: 18 (13 by maintainers)
This is a big problem for us as well
@alexec I understand that you are busy with lots of other issues, just note that the test case above takes not more than 10-15 minutes to perform. In the version 3.3.8 we can still use the PNS executor, but would appreciate if this issue is solved before Emissary will be the only supported executor.
From what I am seeing It’s not necessary or rather it’s only necessary if you have not configured the certificate to your docker image repo as shown above(by post from @vitalyrychkov) in your workflow.
Thank you for your detailed explanation. I think other uses will find this helpful. I think your approach is fine.
Hi Alex, thank you very much for the fix!
With the “latest” tag deployed, I have repeated my test case:
I assume that Emissary is the only supported executor in the latest build and does not recognize the executor-related keywords: containerRuntimeExecutor and containerRuntimeExecutors. I had to remove them from the configmap for the workflow controller to start successfully.
Then I have submitted my test command and got the MESSAGE (found it only in the Argo GUI):
failed to look-up entrypoint/cmd for image "<image_registry_host>/vitaly/cmdtest:latest", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: Get "https://<image_registry_host>/v2/": x509: certificate signed by unknown authorityWe have already had the internal root CA mounted as a volume to the Argo server container, but until now we did not need it for the controller container.
Added the internal root CA as a volume mount to the controller container:
Now the cmdtest pod starts and runs to success, here is the init log:
So, as far as I understand, the issue was the following: unlike PNS, Emissary tries to pull the image manifest from the image repository. In our case our development image registry uses a self-signed certificate and Argo workflow controller could not create TLS connection (which is properly communicated as a message in the latest build). Mounting the CA to the container solves the issue. Am I right?
Would you recommend to keep mounting the CA as a volume to both Argo server and workflow controller as a volume, or is there a better approach?
Sorry. By “latest” I mean ”:latest” image which is built from master branch.
Can you repro on latest?
TEST CASE Create a Dockerfile:
Build the image, e.g.
cmdtest:latestand push to your repoCreate a test workflow to use the built image:
In our Argo installation with Emissary executor the workflow fails with the mentioned error. If I uncomment the command, then the workflow succeeds.