argo-workflows: HTTP Template hangs indefinitely

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

What happened:

• I ran argo submit -n argo --watch httptest.yaml the content from https://argoproj.github.io/argo-workflows/http-template/ . The workflow keeps in ‘Running’ state forever ( more than 5 hours ).

Expected:

• The workflow should succeed in a couple of minutes.

Environment:

% kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:49:09Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/arm64"}

 % argo version
argo: v3.4.3
  BuildDate: 2022-10-31T09:13:13Z
  GitCommit: eddb1b78407adc72c08b4ed6be8f52f2a1f1316a
  GitTreeState: clean
  GitTag: v3.4.3
  GoVersion: go1.18.7
  Compiler: gc
  Platform: darwin/amd64

minikube v1.28.0 on Darwin 13.0 (arm64)

Version

v3.4.3

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

https://argoproj.github.io/argo-workflows/http-template/

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep http-template-8879r

% kubectl logs -n argo deploy/workflow-controller | grep http-template-8879r
time="2022-11-08T02:00:29.184Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.189Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.189Z" level=info msg="Steps node http-template-8879r initialized Running" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.189Z" level=info msg="StepGroup node http-template-8879r-3317262273 initialized Running" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="HTTP node http-template-8879r-1743288300 initialized Pending" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="Workflow step group node http-template-8879r-3317262273 not yet completed" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="Creating TaskSet" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.194Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.207Z" level=info msg="Created Agent pod" namespace=argo podName=http-template-8879r-1340600742-agent workflow=http-template-8879r
time="2022-11-08T02:00:29.207Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.207Z" level=info msg=assessAgentPodStatus namespace=argo podName=http-template-8879r-1340600742-agent
time="2022-11-08T02:00:29.213Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=7906 workflow=http-template-8879r
time="2022-11-08T02:00:39.209Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.210Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=http-template-8879r
time="2022-11-08T02:00:39.210Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.210Z" level=info msg=assessAgentPodStatus namespace=argo podName=http-template-8879r-1340600742-agent
time="2022-11-08T02:00:39.211Z" level=info msg="Workflow step group node http-template-8879r-3317262273 not yet completed" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.211Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.211Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.440Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.441Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=http-template-8879r
time="2022-11-08T02:05:36.441Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.441Z" level=info msg=assessAgentPodStatus namespace=argo podName=http-template-8879r-1340600742-agent
time="2022-11-08T02:05:36.442Z" level=info msg="Workflow step group node http-template-8879r-3317262273 not yet completed" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.442Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.442Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-8879r

Logs from in your workflow’s wait container

% kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=http-template-8879r,workflow.argoproj.io/phase!=Succeeded
error: container wait is not valid for pod http-template-8879r-1340600742-agent