airflow: Tasks with 'none_failed_min_one_success' trigger_rule skipping before Dynamic Task Group is fully expanded
Apache Airflow version
Other Airflow 2 version (please specify below)
If “Other Airflow 2 version” selected, which one?
2.8.3 and 2.9.1
What happened?
If a DAG is run for the first time, a task with a ‘none_failed_min_one_success’ trigger rule is almost immediately skipped before the dependant upstream tasks are complete resulting in:
If I clear the same DAG run, with the process_items task group fully expanded, the DAG completes as expected:
What you think should happen instead?
If the a_end task has a ‘none_failed_min_one_success’ trigger rule it should only be run when its upstream dependant tasks are complete and not in a failed or upstream_failed state and at least one upstream task has succeeded.
How to reproduce
DAG Code:
from datetime import datetime, timedelta
from airflow.decorators import task, task_group
from airflow import DAG, AirflowException
from airflow.operators.empty import EmptyOperator
default_args = {
'owner': 'insight_techops',
'depends_on_past': False,
'start_date': datetime(2019, 1, 1),
'provide_context': True,
'retries': 0,
'retry_delay': timedelta(seconds=30)
}
@task
def parse_csv_schedule():
items_dict = {'A': '1', 'B': '2', 'C': '3', 'D': '4'}
return items_dict
@task_group(group_id="process_items")
def process_items(items_dict: dict):
@task
def retrieve_item_metadata(items: dict):
media_asset = items[0]
print(media_asset)
return media_asset
@task_group(group_id="a_process")
def a_process():
@task.branch(retries=0)
def a_start():
a_handling = 'none'
if a_handling == 'bypass':
return "process_items.a_process.a_bypass"
else:
return "process_items.a_process.a_end"
a_start = a_start()
a_bypass = EmptyOperator(task_id='a_bypass')
a_end = EmptyOperator(task_id='a_end', trigger_rule='none_failed_min_one_success')
a_start >> a_bypass >> a_end
a_start >> a_end
@task
def mark_item_as_done():
try:
print(f"Marking item as Done")
except Exception as error:
raise AirflowException(error)
item_dict = retrieve_item_metadata(items=items_dict)
mark_item_as_done = mark_item_as_done()
a_process = a_process()
item_dict >> a_process >> mark_item_as_done
with DAG(dag_id='af055_Debugger', default_args=default_args, max_active_runs=1, schedule_interval=None, tags=['sales']):
end = EmptyOperator(task_id='end')
items_dict = parse_csv_schedule()
items_dict >> process_items.expand(items_dict=items_dict) >> end
I’m pretty sure it is related to the ‘none_failed_min_one_success’ as if I cause an upstream task within the TaskGroup to fail, the task with the trigger rule in question (and subsequent downstream tasks) are skipped:
Compared to if the trigger rule for the a_end is default:
Operating System
Mac OS 13.6 and Ubuntu 22.0.4
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
Dev: Docker Desktop, Helm, Kubernetes
Prod: EKS, Helm
Anything else?
100% reproducible
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 2
- Comments: 20 (15 by maintainers)
I am observing this same behavior with both
TriggerRule.ONE_SUCCESSandTriggerRule.NONE_FAILED_MIN_ONE_SUCCESSon 2.10.3.I’m stepping back from the assignement for now due to priorities. Anyone is welcome to take this issue. I might reclaim it in the future if no one else does 😃
I’d just like to point out that this issue seems related to #34023 which I’m trying to tackle (for now, without much success). @mateuslatrova - if you’re still working on this issue, I’d be happy if you could also take a look there.
The issue
I will use @RNHTTR 's example DAG to make the explanation simpler. So, we have the following tasks:
When the
inittask instance terminates successfully, both task instances,imsleepyandimawake, are inNonestate and they are also not expanded yet (meaning, theirmap_indexis-1).Then, the scheduler gets the next task instance to be run, which is
imsleepy, checks that its dependencies are met, and expands it into three task instances withNonestate (and map_indexes 0, 1 and 2).Next, the scheduler gets the following task instance to be run, which is
imawake. At this moment, only theinittask instance has finished running. So, when checkingimawaketask instance’s dependencies inside method_evaluate_direct_relativesfromTriggerRuleDepclass, we have the following code to calculate how many upstream tasks it has:https://github.com/apache/airflow/blob/1d7ede78d3440f7752087beccb67c4ebe3c4acdf/airflow/ti_deps/deps/trigger_rule_dep.py#L355-L366
When this code is running for
tibeingimawake’s task instance,task_id_countsis set to 0 by that query, and thenupstreamis also set to 0, which is completely wrong. It should actually be set to 1, because it has 1 upstream task, which isimsleepy.Since
imsleepytask instance’s trigger rule isNONE_FAILED_MIN_ONE_SUCCESSandupstreamis set to 0, the following condition on line 403 is met (becauseskippedis also 0, since no upstream task was skipped), and theimawaketask instance will have its state changed toSKIPPEDeven before being expanded (unwanted behavior).https://github.com/apache/airflow/blob/1d7ede78d3440f7752087beccb67c4ebe3c4acdf/airflow/ti_deps/deps/trigger_rule_dep.py#L400-L404
Now, why did that query return 0? Because the call to
_iter_upstream_conditionsreturned conditiontask_id == imsleepy and map_index < 0and, at this moment, there is no TaskInstance from taskimsleepywithmap_index < 0because it was expanded into three task instances with map_index 0, 1 and 2.So the ROOT CAUSE is: the Database is updated with
imsleepyexpansion data (otherwise, that query above would not result in 0), but the program does not identify that! In this example, inside the_iter_upstream_conditions,TaskInstance.get_relevant_upstream_map_indexesis called and it returns -1, which makes the condition mentioned above be returned by_iter_upstream_conditions.Possible solution
TaskInstance.get_relevant_upstream_map_indexesmethod to cover this edge case.I invite all of you to review and question this explanation to guarantee it is correct, and also, if anyone has suggestions for solutions, they are very welcome!
@mateuslatrova - Great to hear there’s progress with this!
Awesome, thank you @mateuslatrova!
Hi @le-chartreux ! I have already found the issue, and I am working on a solution. Soon I will send an explanation here.
Please do. It would be great if you can confirm that you are able to reproduce the issue.