airflow: Google Stackdriver logging not working

Apache Airflow version

2.5.3

What happened

Using the steps in the documentation within Airflow Helm Chart 1.9 with KubernetesExecutor does not show neither print("This is a print logging message") or

import logging
logger = logging.getLogger("airflow.task")
logger.info("This is a log message")

Authentication is based with GOOGLE_APPLICATION_CREDENTIALS having also the role roles/logging.admin and having the config enabled via values.yaml in the helm and apache-airflow-providers-google==9.0.0 chart with:

config:
  logging:
    remote_logging: 'True'
    remote_base_log_folder: 'stackdriver://airflow-tasks'

The only logs viewable in the UI and both in Google Stackdriver console are related to be only internal, like:

[2023-04-19, 06:39:04 UTC] {taskinstance.py:1309} INFO - Executing <Task(TemplatedPythonOperator): hello-task> on 2023-02-15 00:00:00+00:00 [2023-04-19, 06:39:04 UTC] {standard_task_runner.py:55} INFO - Started process 36 to run task [2023-04-19, 06:40:15 UTC] {local_task_job.py:276} WARNING - State of this instance has been externally set to success. Terminating instance. [2023-04-19, 06:40:15 UTC] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 36. PIDs of all processes in the group: [36] [2023-04-19, 06:40:15 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 36 [2023-04-19, 06:40:16 UTC] {process_utils.py:79} INFO - Process psutil.Process(pid=36, status=‘terminated’, exitcode=0, started=‘06:39:04’) (36) terminated with exit code 0

What you think should happen instead

The Stackdriver remote logging should output the logs in the Web UI and Stackdriver console.

How to reproduce

Use of:

GOOGLE_APPLICATION_CREDENTIALS with role roles/logging.admin
Helm 1.9
KubernetesExecutor
apache-airflow-providers-google==9.0.0
Values Helm in the chart :

config:
  logging:
    remote_logging: 'True'
    remote_base_log_folder: 'stackdriver://airflow-tasks'

Operating System

Helm 1.9

Versions of Apache Airflow Providers

apache-airflow-providers-google==9.0.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Using official Helm Chart 1.9 (also reproducible with 1.8, 1.7)

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project’s Code of Conduct

About this issue

Original URL
State: open
Created a year ago
Comments: 37 (37 by maintainers)

Most upvoted comments

Here’s a note i have from trigger logging dev time:

It rung a bit of a bell and I’m not sure it was actually working. Though I def had it working in my local env by the time of merging the trigger logging change into main. Though I don’t know what exactly was wrong with it. You could look at that commit and might reveal something. Or you might try running locally from main and see what you get. I am gone through end of this weekend. By that time we might have a 2.6 RC you could try

dstandish on Apr 19, 2023