argo-workflows: Flood of "Unable to obtain node for" error messages
Pre-requisites
- I have double-checked my configuration
- I can confirm the issues exists when I tested with
:latest - I’d like to contribute the fix myself (see contributing guide)
What happened/what you expected to happen?
Seeing a lot of these errors:
time="2023-11-02T12:45:44.922Z" level=error msg="was unable to obtain node for " namespace=argo workflow=hello-world-h7bpn
time="2023-11-02T12:45:44.922Z" level=error msg="was unable to obtain node for " namespace=argo workflow=hello-world-7vspx
This was introduced in https://github.com/argoproj/argo-workflows/pull/11665
cc @isubasinghe
Version
latest
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.
Hello world example
Logs from the workflow controller
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}
Logs from in your workflow’s wait container
kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 12
- Comments: 21 (11 by maintainers)
So if the node was not initialized yet, then there shouldn’t be a call to
Nodes.Getat all. The previous code already did that though, meaning the previous code was incorrect. The error message is an indication of a bug somewhere. If we fix the root cause of that bug, then the error should no longer show up.The error message is not the problem, the underlying bug is.
Hi,
I continue in this thread because I have the same problem. I am using KinD with Argoworkflow 3.5.12.
Workflow submit is ok but I guess this error is a symptom of something wrong, and I guess it is my root cause of huge delay before steps to run … Using Kind, I have only one node which is were argoworkflow is running.
Here are some logs (50 seconds probably looking for node instead of running steps) :
May be worth seeing if this still reproduces after https://github.com/argoproj/argo-workflows/pull/12233
Per my previous comments, this message is generally* indicative of a bug somewhere. It can appear in many different places, so context is critical, e.g. if it happens on a certain Workflow every time. Without those details, comments are not very helpful.
*= as Isitha wrote above, sometimes it’s not a bug, but an unconditional log; in which case, the code should be modified to not log and have a comment about why the error is ignored / or not use the code which errors in the first place (e.g. it could be skipped entirely if the node will always be unintialized).I suspect this arises from the dag as mentioned previously, I am now going to find out why we can ignore the error and I will document this. If we can successfully ignore the error, I will remove the log for that particular section.
Very much agreed