airflow: Tasks with 'none_failed_min_one_success' trigger_rule skipping before Dynamic Task Group is fully expanded

Apache Airflow version

Other Airflow 2 version (please specify below)

If “Other Airflow 2 version” selected, which one?

2.8.3 and 2.9.1

What happened?

If a DAG is run for the first time, a task with a ‘none_failed_min_one_success’ trigger rule is almost immediately skipped before the dependant upstream tasks are complete resulting in: image

If I clear the same DAG run, with the process_items task group fully expanded, the DAG completes as expected: image

What you think should happen instead?

If the a_end task has a ‘none_failed_min_one_success’ trigger rule it should only be run when its upstream dependant tasks are complete and not in a failed or upstream_failed state and at least one upstream task has succeeded.

How to reproduce

DAG Code:

from datetime import datetime, timedelta

from airflow.decorators import task, task_group
from airflow import DAG, AirflowException
from airflow.operators.empty import EmptyOperator

default_args = {
    'owner': 'insight_techops',
    'depends_on_past': False,
    'start_date': datetime(2019, 1, 1),
    'provide_context': True,
    'retries': 0,
    'retry_delay': timedelta(seconds=30)
}

@task
def parse_csv_schedule():
    items_dict = {'A': '1', 'B': '2', 'C': '3', 'D': '4'}
    return items_dict

@task_group(group_id="process_items")
def process_items(items_dict: dict):
    @task
    def retrieve_item_metadata(items: dict):
        media_asset = items[0]
        print(media_asset)
        return media_asset

    @task_group(group_id="a_process")
    def a_process():
        @task.branch(retries=0)
        def a_start():
            a_handling = 'none'
            if a_handling == 'bypass':
                return "process_items.a_process.a_bypass"
            else:
                return "process_items.a_process.a_end"
            
        a_start = a_start()
        a_bypass = EmptyOperator(task_id='a_bypass')
        a_end = EmptyOperator(task_id='a_end', trigger_rule='none_failed_min_one_success')
        a_start >> a_bypass >> a_end
        a_start >> a_end

    @task
    def mark_item_as_done():
        try:
            print(f"Marking item as Done")
        except Exception as error:
            raise AirflowException(error)
    
    item_dict = retrieve_item_metadata(items=items_dict)
    mark_item_as_done = mark_item_as_done()
    a_process = a_process()
    item_dict >> a_process >> mark_item_as_done

with DAG(dag_id='af055_Debugger', default_args=default_args, max_active_runs=1, schedule_interval=None, tags=['sales']):
    end = EmptyOperator(task_id='end')
    items_dict = parse_csv_schedule()
    items_dict >> process_items.expand(items_dict=items_dict) >> end

I’m pretty sure it is related to the ‘none_failed_min_one_success’ as if I cause an upstream task within the TaskGroup to fail, the task with the trigger rule in question (and subsequent downstream tasks) are skipped: image

Compared to if the trigger rule for the a_end is default: image

Operating System

Mac OS 13.6 and Ubuntu 22.0.4

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Dev: Docker Desktop, Helm, Kubernetes

Prod: EKS, Helm

Anything else?

100% reproducible

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 2
  • Comments: 20 (15 by maintainers)

Most upvoted comments

I am observing this same behavior with both TriggerRule.ONE_SUCCESS and TriggerRule.NONE_FAILED_MIN_ONE_SUCCESS on 2.10.3.

I’m stepping back from the assignement for now due to priorities. Anyone is welcome to take this issue. I might reclaim it in the future if no one else does 😃

I’d just like to point out that this issue seems related to #34023 which I’m trying to tackle (for now, without much success). @mateuslatrova - if you’re still working on this issue, I’d be happy if you could also take a look there.

The issue

I will use @RNHTTR 's example DAG to make the explanation simpler. So, we have the following tasks:

init >> tg (imsleepy >> imawake)

When the init task instance terminates successfully, both task instances, imsleepy and imawake, are in None state and they are also not expanded yet (meaning, their map_index is -1).

Then, the scheduler gets the next task instance to be run, which is imsleepy, checks that its dependencies are met, and expands it into three task instances with None state (and map_indexes 0, 1 and 2).

Next, the scheduler gets the following task instance to be run, which is imawake. At this moment, only the init task instance has finished running. So, when checking imawake task instance’s dependencies inside method _evaluate_direct_relatives from TriggerRuleDep class, we have the following code to calculate how many upstream tasks it has:

https://github.com/apache/airflow/blob/1d7ede78d3440f7752087beccb67c4ebe3c4acdf/airflow/ti_deps/deps/trigger_rule_dep.py#L355-L366

When this code is running for ti being imawake’s task instance, task_id_counts is set to 0 by that query, and then upstream is also set to 0, which is completely wrong. It should actually be set to 1, because it has 1 upstream task, which is imsleepy.

Since imsleepy task instance’s trigger rule is NONE_FAILED_MIN_ONE_SUCCESS and upstream is set to 0, the following condition on line 403 is met (because skipped is also 0, since no upstream task was skipped), and the imawake task instance will have its state changed to SKIPPED even before being expanded (unwanted behavior).

https://github.com/apache/airflow/blob/1d7ede78d3440f7752087beccb67c4ebe3c4acdf/airflow/ti_deps/deps/trigger_rule_dep.py#L400-L404

Now, why did that query return 0? Because the call to _iter_upstream_conditions returned condition task_id == imsleepy and map_index < 0 and, at this moment, there is no TaskInstance from task imsleepy with map_index < 0 because it was expanded into three task instances with map_index 0, 1 and 2.

So the ROOT CAUSE is: the Database is updated with imsleepy expansion data (otherwise, that query above would not result in 0), but the program does not identify that! In this example, inside the _iter_upstream_conditions, TaskInstance.get_relevant_upstream_map_indexes is called and it returns -1, which makes the condition mentioned above be returned by _iter_upstream_conditions.

Possible solution

  1. Insert a specific (not sure which) condition in the TaskInstance.get_relevant_upstream_map_indexes method to cover this edge case.

I invite all of you to review and question this explanation to guarantee it is correct, and also, if anyone has suggestions for solutions, they are very welcome!

@mateuslatrova - Great to hear there’s progress with this!

Awesome, thank you @mateuslatrova!

Hi @le-chartreux ! I have already found the issue, and I am working on a solution. Soon I will send an explanation here.

Please do. It would be great if you can confirm that you are able to reproduce the issue.