argo-workflows: HTTP Template hangs indefinitely

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

What happened:

Expected:

  • The workflow should succeed in a couple of minutes.

Environment:

% kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:49:09Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/arm64"}
 % argo version
argo: v3.4.3
  BuildDate: 2022-10-31T09:13:13Z
  GitCommit: eddb1b78407adc72c08b4ed6be8f52f2a1f1316a
  GitTreeState: clean
  GitTag: v3.4.3
  GoVersion: go1.18.7
  Compiler: gc
  Platform: darwin/amd64

minikube v1.28.0 on Darwin 13.0 (arm64)

Version

v3.4.3

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

https://argoproj.github.io/argo-workflows/http-template/

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep http-template-8879r

% kubectl logs -n argo deploy/workflow-controller | grep http-template-8879r
time="2022-11-08T02:00:29.184Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.189Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.189Z" level=info msg="Steps node http-template-8879r initialized Running" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.189Z" level=info msg="StepGroup node http-template-8879r-3317262273 initialized Running" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="HTTP node http-template-8879r-1743288300 initialized Pending" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="Workflow step group node http-template-8879r-3317262273 not yet completed" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.190Z" level=info msg="Creating TaskSet" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.194Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.207Z" level=info msg="Created Agent pod" namespace=argo podName=http-template-8879r-1340600742-agent workflow=http-template-8879r
time="2022-11-08T02:00:29.207Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:29.207Z" level=info msg=assessAgentPodStatus namespace=argo podName=http-template-8879r-1340600742-agent
time="2022-11-08T02:00:29.213Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=7906 workflow=http-template-8879r
time="2022-11-08T02:00:39.209Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.210Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=http-template-8879r
time="2022-11-08T02:00:39.210Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.210Z" level=info msg=assessAgentPodStatus namespace=argo podName=http-template-8879r-1340600742-agent
time="2022-11-08T02:00:39.211Z" level=info msg="Workflow step group node http-template-8879r-3317262273 not yet completed" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.211Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:00:39.211Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.440Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.441Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=http-template-8879r
time="2022-11-08T02:05:36.441Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.441Z" level=info msg=assessAgentPodStatus namespace=argo podName=http-template-8879r-1340600742-agent
time="2022-11-08T02:05:36.442Z" level=info msg="Workflow step group node http-template-8879r-3317262273 not yet completed" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.442Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-8879r
time="2022-11-08T02:05:36.442Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-8879r

Logs from in your workflow’s wait container

% kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=http-template-8879r,workflow.argoproj.io/phase!=Succeeded
error: container wait is not valid for pod http-template-8879r-1340600742-agent

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 1
  • Comments: 21 (9 by maintainers)

Most upvoted comments

@mcgrawia The HTTP Template is the main reason I’m still considering Argo workflows. I was looking for a way to “chain” various REST calls, or at least call those in sequence. I was expecting the template to make external a REST call and add the result to its output. It would be nice to have a way for the output of one template to become the input of the next.

@gcsfred2 can you provide the pod events and pod spec for the agent pod? looks like the agent is not coming up.

Can you paste the log from your agent pod (http-template-8879r-1340600742-agent)?