argo-workflows: All workflows fail on EKS 1.20 - ERROR opening annotation file from /argo/podmetadata/annotations

Summary

I upgrade to EKS 1.20. Now when running a workflows with the emissary executor, I get the following:

init time="2021-06-24T11:59:25.250Z" level=info msg="Starting Workflow Executor" executorType=emissary version=v3.1.0
init time="2021-06-24T11:59:25.251Z" level=error msg="ERROR opening annotation file from /argo/podmetadata/annotations"
init panic: open /argo/podmetadata/annotations: no such file or directory
init goroutine 1 [running]:
init github.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.checkErr(0x2463880, 0xc00020e7b0)
init     /go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/root.go:136 +0xf4
init github.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.initExecutor(0x203000)
init     /go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/root.go:101 +0x57a
init github.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.loadArtifacts(0x24b22a0, 0xc00005e018, 0x0, 0x0)
init     /go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/init.go:27 +0x65
init github.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.NewInitCommand.func1(0xc0000e1b80, 0xc000525060, 0x0, 0x2)
init     /go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/init.go:17 +0x3d
init github.com/spf13/cobra.(*Command).execute(0xc0000e1b80, 0xc000525040, 0x2, 0x2, 0xc0000e1b80, 0xc000525040)
init     /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846 +0x2c2
init github.com/spf13/cobra.(*Command).ExecuteC(0xc0000e1080, 0x2258fc8, 0xc00051c0c8, 0xc000893f48)
init     /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
init github.com/spf13/cobra.(*Command).Execute(...)
init     /go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
init main.main()
init     /go/src/github.com/argoproj/argo-workflows/cmd/argoexec/main.go:15 +0x2b

The pod then dies.

Diagnostics

👀 Yes! We need all of your diagnostics, please make sure you add it all, otherwise we’ll go around in circles asking you for it:

What Kubernetes provider are you using? EKS 1.20

What version of Argo Workflows are you running? 3.1.0

What executor are you running? Emissary

Did this work in a previous version? I.e. is it a regression? Worked in EKS 1.19 with workflows 3.1.0

Are you pasting thousands of log lines? That’s too much information.

# Either a workflow that reproduces the bug, or paste you whole workflow YAML, including status, something like:
kubectl get wf -o yaml ${workflow}

# Logs from the workflow controller:
Nothing super out of the ordinary. I did spot this that pops up every few seconds, but it might have been there all along and I hadn't noticed:
`time="2021-06-24T12:02:01.893Z" level=info msg=healthz age=5m0s err="<nil>" instanceID= labelSelector="!workflows.argoproj.io/phase,!workflows.argoproj.io/controller-instanceid" managedNamespace=`

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.