opencensus-python: AzureLogHandler does not seem to work from Celery tasks
Python 3.5.2
Django 2.2.10
celery==4.2.2
opencensus==0.7.7
opencensus-context==0.1.1
opencensus-ext-azure==1.0.2
opencensus-ext-django==0.7.2
opencensus-ext-logging==0.1.0
After a loooong investigation and many experiments with various ways of configuring logging in Celery tasks and django, I am pretty sure that for some reason, the AzureLogHandler does not send any log to Azure from a task running in a Celery worker.
To make sure of that, in my django project, I setup Celery logging by discarding the default Celery logging behaviour (the default is misleading because it overrides the root logger) and instead I use the django logging configuration (from django’s settings.py
), and in addition I manually make sure the celery
logger has both a local file log handler (making sure I capture all log messages locally for debugging this issue) and a AzureLogHandler. You can see how this is configuring in the code snippet below:
import os
import logging
from celery import Celery
from celery.signals import setup_logging
from opencensus.ext.azure.log_exporter import AzureLogHandler
from django.conf import settings
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "edgefolio.settings")
# Create the Celery app (called "api")
app = Celery('api')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
def add_azure_log_handler_to_logger(logger, propagate=True):
"""
Given a logger, add a AzureLogHandler to it
:param logger:
:param propagate:
:return:
"""
if settings.AZURE_APP_INSIGHTS_INSTR_KEY:
formatter = logging.Formatter("[Celery/%(processName)s] %(message)s")
# Azure Handler:
azure_log_handler = AzureLogHandler(
connection_string='InstrumentationKey=%s' % settings.AZURE_APP_INSIGHTS_INSTR_KEY)
azure_log_handler.setFormatter(formatter)
azure_log_handler.setLevel(logging.INFO)
logger.addHandler(azure_log_handler)
logger.setLevel(logging.INFO)
logger.propagate = propagate
return logger
@setup_logging.connect
def setup_loggers(*args, **kwargs):
"""
Using the celery "setup_logging" signal to override and fully define the logging configuration for Celery
:param args:
:param kwargs:
:return:
"""
# Configure Celery logging from the Django settings' logging configuration
from logging.config import dictConfig
from django.conf import settings
dictConfig(settings.LOGGING)
# Test the root logger (configured in django settings to log to Azure as well
logger = logging.getLogger('root')
logger.warning('TRYING LOGGING FROM [%s]' % logger.name)
# Configure the Celery top level logger
logger = logging.getLogger('celery')
# Add a local file log handler to make sure we capture every message locally
logger.addHandler(logging.FileHandler("/data/log/worker/importer.log"))
# In addition, also manually add a AzureLogHandler to it (duplicate with the root's handler)
logger = add_azure_log_handler_to_logger(logger, propagate=False)
# Log a test warning message
logger.warning('TRYING LOGGING FROM [%s]' % logger.name)
# Log a test warning message from a lower-level celery logger
logger = logging.getLogger('celery.task')
logger.warning('TRYING LOGGING FROM [%s]' % logger.name)
# Log a test warning message from a specific django app task logger
logger = logging.getLogger('etl.tasks')
logger.warning('TRYING LOGGING FROM [%s]' % logger.name)
The result of this setup is that the log messages logged from the main celery process are all sent to both the local file and to Azure Monitor without problem, however log messages generated from inside the celery tasks are sent to the local file (proving that the logging configuration above is used in the tasks as well) but NOT to Azure Monitor. It appears only logs from the main Celery process are sent to Azure, but none from the Celery Worker processes.
I wonder if this has to do with the way AzureLogHandler is implemented (using threading and similar wizardry) and the way Celery worker processes run.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 19 (2 by maintainers)
AzureLogHandler doesn’t work with celery for me either. Any update on this?
Good to know I wasn’t the only one facing this. Seems there is no work-around yet.
@SanthoshMedide Can you open up a new issue for this? It seems to be unrelated to this thread.