argo-workflows: Liveness probe fails with 500 and `workflow never reconciled` -- under-resourced k8s apiserver
Pre-requisites
- I have double-checked my configuration
- I can confirm the issues exists when I tested with
:latest - I’d like to contribute the fix myself (see contributing guide)
What happened/what you expected to happen?
This issue explains exactly what happens https://github.com/argoproj/argo-workflows/issues/6351 As it was closed, I’m opening a regression report.
Similar to the parent issue, we are deploying multiple workflows on our kubernetes cluster.
We haven’t enabled Workflow GC/TTL so workflows keep stacking up in the kubernetes etcd.
After a while (when ~100 Workflow CRs are present), we notice Workflows are not being executed anymore. As in the parent issue, we see that status.phase is not being set any more on the Workflow CR.
When checking the /healthz endpoint, we see:
500 workflow never reconciled: workflow-0a9cf5a8-73e4-4d4e-924b-0be6f9c26d51
Because we were using the bitnami helm charts for argo workflows, the Argo Workflows Controller is not restarted because the bitnami helm charts have a bad liveness probe.
Loggingwize, I enabled --debug logging on the workflow controller, but nothing extraordinary is logged. What we do see is that the Watch workflows log line stops printing when the issue occurs:

We think the issue is triggered by the amount of Workflows that are putting stress on K8s etcd + the fact that our kubernetes API server was underdimensioned:

The size of a single workflow manifest is ~100kB.
We are mitigating the issue now by:
- Using the correct liveness probe for the Workflow Controller so it gets automatically restarted when failing.
- Enabling Workflow GC TTL to minimize the load on kubernetes etcd
- Granting more resources to our kubernetes master node.
However, we think the Argo Workflows Controller should be more resilient to a sloppy kubernetes API.
CC @alexec as you were involved in the parent issue.
Version
v3.4.7
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.
Issue is not related to a specific kind of workflow. But by the amount of Workflows present in the kubernetes cluster + the fitness of the Kubernetes API.
Logs from the workflow controller
See main description about what stood out for me in the logs (i.e. the stopping of Watch workflows)
Full logs:
argo_controller.log
Logs from in your workflow’s wait container
$ kubectl logs -n bonzai -c wait -l workflow.argoproj.io/phase!=Succeeded
No resources found in bonzai namespace.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 3
- Comments: 16 (4 by maintainers)
I found a solution for my problem. It seems that argowf controller was not able to reconcile some workflows from the queue, so I had to go and delete than manually. I guess this happened because my pod was understimated for the volume of workflows we were having. Só I change the replicas from 1 to 2. And now the liveness probe is responding with ok again.