argo-workflows: Changes in docker image entrypoint are ignored when using same tag

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I used a docker image in a template, without specifying a command. The field reference documentation says:

The docker image’s ENTRYPOINT is used if this is not provided.

And that worked when running the image for the first time.

Now I made some changes to the image, especially I changed its ENTRYPOINT. Everything else is as before; especially the image was pushed under the same tag. I use imagePullPolicy: Always, so I expect the new image to run. Inspecting the generated pod, I can see that the new image was indeed pulled from the registry (it matches the sha256 sum).

However, also in the generated pod, I can still see the old entrypoint:

  command:
    - /var/run/argo/argoexec
    - emissary
    - --loglevel
    - info
    - --log-format
    - text
    - --
    - OLD_ENTRY_POINT

I have no idea where it came from, but as this command is supposedly generated by argo, I guess it caches this information somewhere. It works when I restart the workflow controller.

Version

v3.4.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

metadata:
  name: test
  namespace: default
spec:
  templates:
    - name: test
      container:
        name: test
        image: docker.io/busybox:latest # well, you likely need a private repo here and some meaningful image to test changes
        args:
          - 'ls'
          - '/'
  entrypoint: test

Logs from the workflow controller

not applicable

Logs from in your workflow’s wait container

not applicable

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 1
  • Comments: 18 (5 by maintainers)

Most upvoted comments

@tooptoop4 Because snapshot releases in a dev environment? For the record: I don’t want to start a discussion here on how everyone should run her/his infrastructure and neither should you.

Another workaround is restarting the Argo workflow controller, since the cache seems to be only in-memory and can thus be reset that way.

Interesting, that would suggest it’s the Controller’s cache that is the problem definitively. That sounds like it may be using the image:tag as a cache key, but that is actually not a unique identifier and so can be affected by collisions. If that is the case, it should use the image’s digest SHA instead

why not using immutable tags 😱

and the description for command says “The docker image’s ENTRYPOINT is used if this is not provided.”

that is correct, here’s the exact code block for reference

With the current behavior this is not true.

It seems like a stale cache, so it’s using the old ENTRYPOINT

We switched to emissary executor so the command must be specified explicitly.

I believe this used to be true prior to #8345. there are some cases where that fails as well (for instance, Docker manifest v1 of old images) and so a command must be given in those cases as well.

Ran into the same issue (entrypoint changed in image for same tag, command is cached). Another workaround is restarting the Argo workflow controller, since the cache seems to be only in-memory and can thus be reset that way.

@JPZ13 now that I am aware, I can live with it. Another workaround would be to just specify the command. Speaking for me, I normally do not just change the entrypoint and leave everything else as-is.

I am quite sure that it is strictly happening with argo. I just stumbled across #8345 and I my guess is, that the cache implemented in https://github.com/alexec/argo-workflows/blob/master/workflow/controller/entrypoint/cache_index.go (introduced in this PR) allows for exactly this behavior.

To clarify (@terrytangyuan): it seems that the documentation is outdated. The PR (which was merged in v3.4) specifically says

This PR automatically looks up the command and entrypoint.

@terrytangyuan ah I see now. I was not aware of that.

Then again, I wonder why it kind-of works anyway, opposed to the documentation. Or in other words: why did somebody spend time for building this functionality instead of just throwing an error to the user? From my point of view, this would be better instead of offering the feature but with bugs 😉

However, also in the generated pod, I can still see the old entrypoint

You are looking at the wrong container command. Check your main container.