argo-workflows: Flood of "Unable to obtain node for" error messages

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :latest
  • I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Seeing a lot of these errors:

time="2023-11-02T12:45:44.922Z" level=error msg="was unable to obtain node for " namespace=argo workflow=hello-world-h7bpn
time="2023-11-02T12:45:44.922Z" level=error msg="was unable to obtain node for " namespace=argo workflow=hello-world-7vspx

This was introduced in https://github.com/argoproj/argo-workflows/pull/11665

cc @isubasinghe

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

Hello world example

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow’s wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 12
  • Comments: 21 (11 by maintainers)

Most upvoted comments

We should check whether these errors are necessary. Maybe node was just not initialized yet.

So if the node was not initialized yet, then there shouldn’t be a call to Nodes.Get at all. The previous code already did that though, meaning the previous code was incorrect. The error message is an indication of a bug somewhere. If we fix the root cause of that bug, then the error should no longer show up.

The error message is not the problem, the underlying bug is.

Hi,

I continue in this thread because I have the same problem. I am using KinD with Argoworkflow 3.5.12.

Workflow submit is ok but I guess this error is a symptom of something wrong, and I guess it is my root cause of huge delay before steps to run … Using Kind, I have only one node which is were argoworkflow is running.

Here are some logs (50 seconds probably looking for node instead of running steps) :

time="2024-10-31T13:01:17.855Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T13:01:27.783Z" level=error msg="was unable to obtain node for rinex-ingester-lb5wn-2166136261" namespace=commanding workflow=rinex-ingester-lb5wn
time="2024-10-31T13:01:27.784Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T13:01:37.802Z" level=error msg="was unable to obtain node for rinex-ingester-lb5wn-2166136261" namespace=commanding workflow=rinex-ingester-lb5wn
time="2024-10-31T13:01:37.803Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T13:01:47.867Z" level=error msg="was unable to obtain node for rinex-ingester-lb5wn-2166136261" namespace=commanding workflow=rinex-ingester-lb5wn
time="2024-10-31T13:01:47.872Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T13:01:47.872Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T13:01:47.872Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T13:01:57.917Z" level=error msg="was unable to obtain node for rinex-ingester-lb5wn-2166136261" namespace=commanding workflow=rinex-ingester-lb5wn
time="2024-10-31T13:01:57.918Z" level=debug msg="unresolved is allowed " error=unresolved
time="2024-10-31T13:01:57.918Z" level=debug msg="unresolved is allowed " error=unresolved

May be worth seeing if this still reproduces after https://github.com/argoproj/argo-workflows/pull/12233

Per my previous comments, this message is generally* indicative of a bug somewhere. It can appear in many different places, so context is critical, e.g. if it happens on a certain Workflow every time. Without those details, comments are not very helpful.

* = as Isitha wrote above, sometimes it’s not a bug, but an unconditional log; in which case, the code should be modified to not log and have a comment about why the error is ignored / or not use the code which errors in the first place (e.g. it could be skipped entirely if the node will always be unintialized).

I suspect this arises from the dag as mentioned previously, I am now going to find out why we can ignore the error and I will document this. If we can successfully ignore the error, I will remove the log for that particular section.

So if the node was not initialized yet, then there shouldn’t be a call to Nodes.Get at all. The previous code already did that though, meaning the previous code was incorrect.

Very much agreed