argo-workflows: Getting `ResourceExhausted EOF` when running `argo wait` for too long

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Dear all,

We are running some rather big (> 50 min) workflows in our CI pipelines on Github testing out some of our argo-based ML training.

Only after upgrading argo to v3.4.10 lately we’ve gotten some ResourceExhausted EOF error when waiting for the jobs to succeed: level=fatal msg="rpc error: code = ResourceExhausted desc = EOF"

We figured that at some point after 50 mins the argo server would respond badly to our argo wait which should substantially wait an infinite time until the workflow either succeeds or fails. We looked for the specific error response but didn’t find anything. Finally we ended up running the argo wait in an try-except loop like

# submit argo workflow
  workflow_name=$(
    argo submit \
      --from "workflows/$WORKFLOW_NAME" \
      -n namespaceAbc \
      -o name
  )

  # wait for argo wait (retries) to finish
  retries=3
  for ((i=1; i<=retries; i++)); do
    echo "Waiting for workflow to complete (Attempt $i)..."
    # Run the argo wait command and capture the exit status
    exit_status=0
    argo wait "$workflow_name" -n namespaceAbc || exit_status=1
    # Check if the workflow was successful (exit status 0) or if we've reached the maximum retries
    if [ $exit_status -eq 0 ]; then
        echo "Workflow completed successfully."
        break  # Exit the loop
    elif [ "$i" -eq $retries ]; then
        echo "Maximum retries reached. Workflow did not complete successfully."
        exit 1  # Exit with an error code
    else
        echo "Workflow did not complete successfully. Retrying in 30 seconds..."
        sleep 30  # Wait for a period before the next retry
    fi
  done

  # sleep for 30 seconds to be sure that metadata is updated
  echo "Sleeping for 30 seconds to be sure that metadata is updated"
  sleep 30

  # get workflow status from json
  workflow_status=$(
    argo get "$workflow_name" \
      -n namespaceAbc \
      -o json | jq -r '.status.phase'
  )

  # if workflow status is not succeeded, exit with 1
  if [ "$workflow_status" != "Succeeded" ]; then
    echo "Workflow $workflow_name failed with status $workflow_status"
    exit 1
  fi

which would end in the CI job two times receiving the timeout EOF error and only in the third argo wait to finally reach the end of the job with the awaited status succeeded.

CC / @Depaccu

Version

v3.4.10

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

None

Logs from the workflow controller

None

Logs from in your workflow’s wait container

None

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 11
  • Comments: 17 (5 by maintainers)

Most upvoted comments

If you’re running the 3.5.7 server you should upgrade to 3.5.8, it has a memory corruption bug. What happens is uncertain beyond crashing, it could be fine or it could be causing issues.

We started encountering this issue after upgrading Argo Workflows from 3.4.10 to 3.5.7 and argo cli from 3.3.8 to 3.5.7 It reproduces rather frequently for any workflows running >1h and the error appears roughly every 30 - 50 minutes, we have worked around it by implementing a loop similar to the one mentioned in the issue description. e.g.

time="2024-06-18T07:06:59 UTC" level=fatal msg="rpc error: code = ResourceExhausted desc = EOF"
time="2024-06-18T07:43:08 UTC" level=fatal msg="rpc error: code = ResourceExhausted desc = EOF"
time="2024-06-18T08:26:06 UTC" level=fatal msg="rpc error: code = ResourceExhausted desc = EOF"
<workflow> Succeeded at 2024-06-18 08:56:07 +0000 UTC

We were not encountering this issue at all for combination: Argo Workflows 3.4.10 + argo cli 3.3.8.

Your argo server may well be crashing on 3.5.7, see #13166. See if 3.5.8 helps with that.

Facing a similar issue on version 3.4.9 Tried the latest version 3.5.4 but issue still exist

What version did you update from? That would be very helpful in narrowing down a potential regression

Before we was on 3.4.4.

Any logs from argo server?