argo-workflows: Workflow with Http call stuck on Pending State

Checklist

  • Double-checked my configuration.
  • Tested using the latest version.
  • Used the Emissary executor.

Summary

I was trying to do an Http Call from Workflow. But workflow got stuck in Pending State. I noticed this bug - https://github.com/argoproj/argo-workflows/pull/7084 Thought this is fixed. But I am seeing that now also. I am on v3.3.0

What version are you running? v3.3.0

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.


metadata:
  name: delightful-python
  namespace: default
  labels:
    example: 'true'
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: get-google-homepage
            template: http
            arguments:
              parameters: [{name: url, value: "https://www.google.com"}]
    - name: http
      inputs:
        parameters:
          - name: url
      http:
        timeoutSeconds: 20 # Default 30
        url: "{{inputs.parameters.url}}"
        method: "GET" 
        headers:
          - name: "x-header-name"
            value: "test-value"
        successCondition: "response.body contains \"google\""  
        body: "test body"  

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow} 

time="2022-04-01T09:29:28.761Z" level=info msg="Processing workflow" namespace=default workflow=delightful-python
time="2022-04-01T09:29:28.762Z" level=info msg=updateAgentPodStatus namespace=default workflow=delightful-python
time="2022-04-01T09:29:28.763Z" level=info msg=assessAgentPodStatus namespace=default podName=delightful-python-1340600742-agent
time="2022-04-01T09:29:28.764Z" level=info msg="Workflow step group node delightful-python-4228632240 not yet completed" namespace=default workflow=delightful-python
time="2022-04-01T09:29:28.764Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=delightful-python
time="2022-04-01T09:29:28.764Z" level=info msg=reconcileAgentPod namespace=default workflow=delightful-python

# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${delightful-python},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Message from the maintainers:

Impacted by this bug? Give it a đź‘Ť. We prioritise the issues with the most đź‘Ť.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 7
  • Comments: 15 (4 by maintainers)

Most upvoted comments

@clementlecorre @yeongu-dev I use the Helm chart to deploy ArgoWorkflows and serviceaccounts. I wanted to use the http template but got "Watch workflowtasksets 403" error in the pod too. I noticed there the default serviceaccount was used. Specifying the right serviceaccount solves the issue in my case : argo submit --serviceaccount argo-workflow

@clementlecorre did you make any progress on this?

I have created the agent role and added a rolebinding for my service account and I still see always-pending HTTP templates with no errors logged.

My agent pod appears to be in permanent “init” state:

http-condition-zrw6k-1340600742-agent 0/1 Init:0/1 0 13m

EDIT: after using kubectl describe pod http-condition-zrw6k-1340600742-agent to inspect the pod status I saw the following event

Warning FailedMount 4s (x6 over 19s) kubelet MountVolume.SetUp failed for volume “kube-api-access-7gslv” : secret “default.service-account-token” not found

I followed this process (with added -n argo argument) to create a token secret and then the HTTP template was able to execute as expected. I did not see this step described in any of the docs linked from the HTTP template docs.