argo-workflows: Getting `ResourceExhausted EOF` when running `argo wait` for too long
Pre-requisites
- I have double-checked my configuration
- I can confirm the issues exists when I tested with
:latest - I’d like to contribute the fix myself (see contributing guide)
What happened/what you expected to happen?
Dear all,
We are running some rather big (> 50 min) workflows in our CI pipelines on Github testing out some of our argo-based ML training.
Only after upgrading argo to v3.4.10 lately we’ve gotten some ResourceExhausted EOF error when waiting for the jobs to succeed: level=fatal msg="rpc error: code = ResourceExhausted desc = EOF"
We figured that at some point after 50 mins the argo server would respond badly to our argo wait which should substantially wait an infinite time until the workflow either succeeds or fails. We looked for the specific error response but didn’t find anything. Finally we ended up running the argo wait in an try-except loop like
# submit argo workflow
workflow_name=$(
argo submit \
--from "workflows/$WORKFLOW_NAME" \
-n namespaceAbc \
-o name
)
# wait for argo wait (retries) to finish
retries=3
for ((i=1; i<=retries; i++)); do
echo "Waiting for workflow to complete (Attempt $i)..."
# Run the argo wait command and capture the exit status
exit_status=0
argo wait "$workflow_name" -n namespaceAbc || exit_status=1
# Check if the workflow was successful (exit status 0) or if we've reached the maximum retries
if [ $exit_status -eq 0 ]; then
echo "Workflow completed successfully."
break # Exit the loop
elif [ "$i" -eq $retries ]; then
echo "Maximum retries reached. Workflow did not complete successfully."
exit 1 # Exit with an error code
else
echo "Workflow did not complete successfully. Retrying in 30 seconds..."
sleep 30 # Wait for a period before the next retry
fi
done
# sleep for 30 seconds to be sure that metadata is updated
echo "Sleeping for 30 seconds to be sure that metadata is updated"
sleep 30
# get workflow status from json
workflow_status=$(
argo get "$workflow_name" \
-n namespaceAbc \
-o json | jq -r '.status.phase'
)
# if workflow status is not succeeded, exit with 1
if [ "$workflow_status" != "Succeeded" ]; then
echo "Workflow $workflow_name failed with status $workflow_status"
exit 1
fi
which would end in the CI job two times receiving the timeout EOF error and only in the third argo wait to finally reach the end of the job with the awaited status succeeded.
CC / @Depaccu
Version
v3.4.10
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.
None
Logs from the workflow controller
None
Logs from in your workflow’s wait container
None
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 11
- Comments: 17 (5 by maintainers)
If you’re running the 3.5.7 server you should upgrade to 3.5.8, it has a memory corruption bug. What happens is uncertain beyond crashing, it could be fine or it could be causing issues.
We started encountering this issue after upgrading Argo Workflows from 3.4.10 to 3.5.7 and argo cli from 3.3.8 to 3.5.7 It reproduces rather frequently for any workflows running >1h and the error appears roughly every 30 - 50 minutes, we have worked around it by implementing a loop similar to the one mentioned in the issue description. e.g.
We were not encountering this issue at all for combination: Argo Workflows 3.4.10 + argo cli 3.3.8.
Your argo server may well be crashing on 3.5.7, see #13166. See if 3.5.8 helps with that.
Facing a similar issue on version 3.4.9 Tried the latest version 3.5.4 but issue still exist
Before we was on
3.4.4.Any logs from argo server?