airflow: Missing Metrics After Migrating from StatsD to OpenTelemetry
Apache Airflow version
2.10.2
If “Other Airflow 2 version” selected, which one?
No response
What happened?
I’ve recently migrated from StatsD to OpenTelemetry and am now sending metrics to Splunk Observability.
However, I’ve noticed that some metrics are now missing after this change, such as operator_successes, operator_failures, and some custom metrics that I used to see when using StatsD with the Python stats package in my DAGs.
The common factor among the missing metrics is that their descriptions in the Airflow documentation mention “Metric with xyz tagging.” I’m not sure if this is relevant.
What you think should happen instead?
All metrics, including operator_successes, operator_failures, and custom metrics, should be available as they were with StatsD.
How to reproduce
- Migrate from StatsD to OpenTelemetry.
- Send metrics to Splunk Observability.
- Observe the missing metrics such as
operator_successes,operator_failures, andcustom metrics.
Operating System
Azure AKS linux
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project’s Code of Conduct
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 6
- Comments: 21 (16 by maintainers)
Maybe a good way is to refactor the code such that resource attributes to be declared as global singleton and every get otel logger function is not recreating that all the time, since the resource attributes are const and should not alter.
@dannyl1u is looking into it
Hi all, I wanted to switch our Airflow to use OTEL and run into the same issue. I debugged into the issue and found:
metrics like “ti.start” and “ti.finish” are exported in the worker context with label dag_id and task_id. The metrics are only available during the time where the task is running. Looks metrics are gone after task finished and then the metric is removed after ~ 5min from the OtelCollector. Maybe because the metrics like “ti.start” are exported in the worker context an the OtelLogger is gone if the worker task is finished?
I´m not sure if singleton instance can solve the issue as Airflow is able to handle multiple workers, schedulers and … For me it looks like the metrics which have the same labels are overwriting each other if they are exported by two different Pods (E.g. 2 workers). So not possible to increase a counter like “ti.start” with label “dag_id” and “task_id” if 2 dag_runs are running in parallel on different workers. Expecting: airflow_ti_start{dag_id=“dag1”, task_id=“task1”} 2 but getting: airflow_ti_start{dag_id=“dag1”, task_id=“task1”} 1
Two main issue to solve:
I write here to start the discussion again and get also some feedback from you all to improve the thinks. Not sure about the best way to fix this issue. We need to solve both points to get usable metrics via Otel.
Started already to improve Airflow Otel implementation a different point https://github.com/apache/airflow/pull/46510
((That last comment only applies to custom metrics in the DAG not getting emitted and would not explain “core” metrics being dropped))
Can confirm I’ve been able to reproduce this bug in version 3.0.0