airflow: Task is not able to be run - k8s Celery workers with stale resources
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
We are facing a quite interesting problem with our celery workers. We run on AWS EKS (m-series nodes running Bottlerocket OS), Airflow v2.6.2.
When the amount of running tasks is high (close to our max cap), we notice an increase in the celery workers memory usage followed by a stale period. At the same time, CPU utilization goes to zero.
During this time, the worker container seems to freeze (no logs are produced). Of the tasks that were running, almost all of them failed with the only log being:
[2023-08-31, 13:20:20 UTC] {taskinstance.py:1093} INFO - Dependencies not met for <TaskInstance: ***** scheduled__2023-08-31T10:00:00+00:00 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
[2023-08-31, 13:20:20 UTC] {local_task_job_runner.py:154} INFO - Task is not able to be run
It is important to note that this problem happens sometimes, but not every time. Which makes it even more confusing. Also, workers pods are not killed in the process, nor do they throw any exceptions.
Some of our configurations are:
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG = 32
AIRFLOW__CORE__PARALLELISM = 64
AIRFLOW__CELERY__WORKER_CONCURRENCY = 12
We are running at the moment with 5 celery workers, with no CPU or memory limits. The scheduler seems fine during those periods, producing regular logs. The only thing I can think of at the moment is that its probably due to a shortage of resources in the celery workers? However the worker concurrency limit should prevent a high usage of CPU/memory. Any ideas will be appreciated.
Some executor metrics on the same time period:
What you think should happen instead
CPU and Memory resources should not go stale. Also, workers should keep logging as usual, instead of “freezing”.
How to reproduce
Deploy through Helm chart, using the CeleryExecutor. Run the maximum number of allowed tasks at the same time. However, the problem is not guaranteed to happen.
Operating System
Bottlerocket OS
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
AWS EKS with k8s v1.26.1 - Nodes (m-series) using Bottlerocket OS.
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 3
- Comments: 19 (7 by maintainers)
@alokshenoy Does setting following help like it helped @keysersoza :
No. I do not see anyone working on it nor providing a solution. But the log above suggests that this is caused by so called “mini scheduler” which is enabled by this configuration https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#schedule-after-task-execution - likely disabling it will mitigate the problem (at the expense of slightly increased latency between tasks.
Can everyone in this thread make an experiment, disable it and report back here - at least this will give some clue on where the problems might happen.
cc: @kaxil @ephraimbuddy -> problems similar to that one keep on appearing, I think there is some kind of deadlock/resource contention with mini scheduler in some specific cases. Might be worth looking at by someone in your teams,
Also facing this issue. Any solution yet?
@potiuk the “cripticity” is already in the lack of logs in Airflow:
The circumstances are not special, just sporadic failures with the logs above. The problem is indeed that I have no details whatsoever.