airflow: Missing Metrics After Migrating from StatsD to OpenTelemetry

Apache Airflow version

2.10.2

If “Other Airflow 2 version” selected, which one?

No response

What happened?

I’ve recently migrated from StatsD to OpenTelemetry and am now sending metrics to Splunk Observability.

However, I’ve noticed that some metrics are now missing after this change, such as operator_successes, operator_failures, and some custom metrics that I used to see when using StatsD with the Python stats package in my DAGs.

The common factor among the missing metrics is that their descriptions in the Airflow documentation mention “Metric with xyz tagging.” I’m not sure if this is relevant.

What you think should happen instead?

All metrics, including operator_successes, operator_failures, and custom metrics, should be available as they were with StatsD.

How to reproduce

  • Migrate from StatsD to OpenTelemetry.
  • Send metrics to Splunk Observability.
  • Observe the missing metrics such as operator_successes, operator_failures, and custom metrics.

Operating System

Azure AKS linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 6
  • Comments: 21 (16 by maintainers)

Most upvoted comments

Maybe a good way is to refactor the code such that resource attributes to be declared as global singleton and every get otel logger function is not recreating that all the time, since the resource attributes are const and should not alter.

@dannyl1u is looking into it

Hi all, I wanted to switch our Airflow to use OTEL and run into the same issue. I debugged into the issue and found:

  1. metrics like “ti.start” and “ti.finish” are exported in the worker context with label dag_id and task_id. The metrics are only available during the time where the task is running. Looks metrics are gone after task finished and then the metric is removed after ~ 5min from the OtelCollector. Maybe because the metrics like “ti.start” are exported in the worker context an the OtelLogger is gone if the worker task is finished?

  2. I´m not sure if singleton instance can solve the issue as Airflow is able to handle multiple workers, schedulers and … For me it looks like the metrics which have the same labels are overwriting each other if they are exported by two different Pods (E.g. 2 workers). So not possible to increase a counter like “ti.start” with label “dag_id” and “task_id” if 2 dag_runs are running in parallel on different workers. Expecting: airflow_ti_start{dag_id=“dag1”, task_id=“task1”} 2 but getting: airflow_ti_start{dag_id=“dag1”, task_id=“task1”} 1

Two main issue to solve:

  1. Any idea how to make the OtelLogger static for one worker pod as the tasks are executed e.g. with the StandardTaskRunner, but then we still have to solve 2)?
  2. How to get trust able metrics in multi Pod deployments.

I write here to start the discussion again and get also some feedback from you all to improve the thinks. Not sure about the best way to fix this issue. We need to solve both points to get usable metrics via Otel.

Started already to improve Airflow Otel implementation a different point https://github.com/apache/airflow/pull/46510

((That last comment only applies to custom metrics in the DAG not getting emitted and would not explain “core” metrics being dropped))

Can confirm I’ve been able to reproduce this bug in version 3.0.0