airflow: Task is not able to be run - k8s Celery workers with stale resources

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

We are facing a quite interesting problem with our celery workers. We run on AWS EKS (m-series nodes running Bottlerocket OS), Airflow v2.6.2.

When the amount of running tasks is high (close to our max cap), we notice an increase in the celery workers memory usage followed by a stale period. At the same time, CPU utilization goes to zero. image

During this time, the worker container seems to freeze (no logs are produced). Of the tasks that were running, almost all of them failed with the only log being:

[2023-08-31, 13:20:20 UTC] {taskinstance.py:1093} INFO - Dependencies not met for <TaskInstance: ***** scheduled__2023-08-31T10:00:00+00:00 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
[2023-08-31, 13:20:20 UTC] {local_task_job_runner.py:154} INFO - Task is not able to be run

It is important to note that this problem happens sometimes, but not every time. Which makes it even more confusing. Also, workers pods are not killed in the process, nor do they throw any exceptions.

Some of our configurations are:

AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG = 32
AIRFLOW__CORE__PARALLELISM = 64
AIRFLOW__CELERY__WORKER_CONCURRENCY = 12

We are running at the moment with 5 celery workers, with no CPU or memory limits. The scheduler seems fine during those periods, producing regular logs. The only thing I can think of at the moment is that its probably due to a shortage of resources in the celery workers? However the worker concurrency limit should prevent a high usage of CPU/memory. Any ideas will be appreciated.

Some executor metrics on the same time period: image image

What you think should happen instead

CPU and Memory resources should not go stale. Also, workers should keep logging as usual, instead of “freezing”.

How to reproduce

Deploy through Helm chart, using the CeleryExecutor. Run the maximum number of allowed tasks at the same time. However, the problem is not guaranteed to happen.

Operating System

Bottlerocket OS

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

AWS EKS with k8s v1.26.1 - Nodes (m-series) using Bottlerocket OS.

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 3
  • Comments: 19 (7 by maintainers)

Most upvoted comments

@alokshenoy Does setting following help like it helped @keysersoza :

  celery:
    task_publish_max_retries: 5
  celery_broker_transport_options:
    visibility_timeout: 86400

No. I do not see anyone working on it nor providing a solution. But the log above suggests that this is caused by so called “mini scheduler” which is enabled by this configuration https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#schedule-after-task-execution - likely disabling it will mitigate the problem (at the expense of slightly increased latency between tasks.

Can everyone in this thread make an experiment, disable it and report back here - at least this will give some clue on where the problems might happen.

cc: @kaxil @ephraimbuddy -> problems similar to that one keep on appearing, I think there is some kind of deadlock/resource contention with mini scheduler in some specific cases. Might be worth looking at by someone in your teams,

Also facing this issue. Any solution yet?

I have been seeing this error appearing after upgrading to 2.7.3, also using the Celery Executor on K8s. However, I don’t see that pattern when looking at resource consumption. I wonder what “Dependencies not met …” means

I think we miss some details but if you can provide mroe @babaMar, logs, explanation what you do, circusmstances where it happens, it might make it easier for someone to diagnose it. Without it and with cryptic " Dependencies not met …" comment it neither helps nor explains what your problem is. And it might be different propblem, so what I suggest is to open a discussion where you will explain everything you see and experience. The more detailed it is, the higher chance that someeone will decide to spend their time on it and maybe even diagnose it.

@potiuk the “cripticity” is already in the lack of logs in Airflow:

[2023-12-12, 08:03:01 UTC] {taskinstance.py:1149} INFO - Dependencies not met for <TaskInstance: ***__2023-12-11T06:20:00+00:00 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
[2023-12-12, 08:03:01 UTC] {local_task_job_runner.py:154} INFO - Task is not able to be run

The circumstances are not special, just sporadic failures with the logs above. The problem is indeed that I have no details whatsoever.