airflow: Task fails and cannot read logs. Invalid URL 'http://:8793/log/...': No host supplied

Apache Airflow version

Other Airflow 2 version (please specify below)

If “Other Airflow 2 version” selected, which one?

2.10.1

What happened?

I’m having an issue with an airflow instance where a task fails and I cannot read the logs.

Logs:

*** Could not read served logs: Invalid URL 'http://:8793/log/dag_id=my_dag/run_id=dynamic__apple_3_my_dag_cb353081__2024-09-09T14:41:22.596199__f73c5571719e4f35bf195ded40e5e25b/task_id=cleanup_temporary_directory/attempt=1.log': No host supplied

Event logs:

Executor CeleryExecutor(parallelism=128) reported that the task instance <TaskInstance: my_dag.cleanup_temporary_directory dynamic__apple_3_my_dag_cb353081__2024-09-09T14:41:22.596199__f73c5571719e4f35bf195ded40e5e25b [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally

Additionally I checked the logs directory for the dag_id/run_id and it’s missing the respective task_id folder.

What you think should happen instead?

I should be able to access the logs.

How to reproduce

Not sure how to.

Operating System

Ubuntu 24.04 LTS

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

Deployed with docker-compose on Docker Swarm setup on 2 VMs.

Anything else?

Additionally I checked the logs directory for the dag_id/run_id and it’s missing the respective task_id folder.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 36
  • Comments: 53 (7 by maintainers)

Most upvoted comments

same issue for us when upgrading to 2.10.2

We are facing a similar issue in versions: 2.10.2 and 2.10.4 in EKS. I noticed that when we had our workers in a Statefulset, they return the full FQDN: airflow-worker-0.airflow-worker.airflow.cluster.local, whereas when they are deployed in a Deployment, they only return: airflow-worker-xxxxxxxx-yyyy. For us, changing hostname_callable to airflow.utils.net.get_host_ip_address fixed the issue. This might be something nice to set automatically when worker’s persistence is disabled in the HelmChart?

I have the same issue with 2.10.0, using the CeleryExecutor. It worked before I upgrading from version 2.9.0 to 2.10.0.

*** Could not read served logs: Invalid URL 'http://:8793/log/dag_id=service_stop/run_id=manual__2024-09-18T09:42:54+09:00/task_id=make_accountlist_task/attempt=1.log': No host supplied

eventlog

Executor CeleryExecutor(parallelism=6) reported that the task instance <TaskInstance: service_stop.make_accountlist_task manual__2024-09-18T09:42:54+09:00 [queued]> finished with state failed, but the task instance's state attribute is queued. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally

Scheduler has a error log at the same hour as eventlog.

[2024-09-18T00:43:18.036+0000] {celery_executor.py:291} ERROR - Error sending Celery task: module 'redis' has no attribute 'client'
Celery Task ID: TaskInstanceKey(dag_id='service_stop', task_id='make_accountlist_task', run_id='manual__2024-09-18T09:42:54+09:00', try_number=1, map_index=-1)
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 220, in send_task_to_executor
    result = task_to_run.apply_async(args=[command], queue=queue)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/task.py", line 594, in apply_async
    return app.send_task(
           ^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 797, in send_task
    with self.producer_or_acquire(producer) as P:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 932, in producer_or_acquire
    producer, self.producer_pool.acquire, block=True,
              ^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 1354, in producer_pool
    return self.amqp.producer_pool
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/amqp.py", line 591, in producer_pool
    self.app.connection_for_write()]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 829, in connection_for_write
    return self._connection(url or self.conf.broker_write_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/base.py", line 880, in _connection
    return self.amqp.Connection(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/connection.py", line 201, in __init__
    if not get_transport_cls(transport).can_parse_url:
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/__init__.py", line 91, in get_transport_cls
    _transport_cache[transport] = resolve_transport(transport)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/__init__.py", line 76, in resolve_transport
    return symbol_by_name(transport)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/airflow/.local/lib/python3.12/site-packages/kombu/transport/redis.py", line 282, in <module>
    class PrefixedRedisPipeline(GlobalKeyPrefixMixin, redis.client.Pipeline):
                                                      ^^^^^^^^^^^^
AttributeError: module 'redis' has no attribute 'client'

got the same behavior since upgrading from version 2.9.3 to 2.10.1. We are using LocalExecutor

Finally found the issue when i trie to upgrade my docker airflow 2 to airflow 3.

For the record im using LocalExecutor, so it wasnt neither redis nor celery causing the issue since i dont need those.

In my case my base_url config in airflow.cfg was set to http://localhost:8080 and it was working fine in Airflow 2 but in Airflow 3 i had to update it by using my docker hostname so in my case it became http://webserver:8080 and the issue was finally gone 😎

Hope this will help some of you !

We are facing the same issue on 2.10.4

We have upgraded on 2.10.1 like @andrew-stein-sp and we could reproduce the same behavior

having the same issue with 2.10.1 in k8s, using the CeleryKubernetesExecutor.

Could this be related to the inheritance issue that was discussed in https://github.com/apache/airflow/issues/41891?

I am running 3.0.1 using CeleryExecutor with valkey (redis) as the message broker and postgres as the result backend. It appears the task instance at the time of building the log url does not have the hostname (ti.hostname link) populated. Not sure if this is just a symptom of a larger issue. This is my first time setting up and trying to run airflow.

Some further testing- I spun up another celery worker using the same credentials as configured for airflow.cfg and did a simple proof of concept using a separate celery task and that succeeded. So it is definitely something within airflow as opposed to my valkey/redis setup/config.

I use docker compose to run airflow and had the same problem for all dags. What I did was stopping all dags, mark all task which were up for retry as failed, and rebuild all containers related to airflow. Now I can run my dags.

I am running 2.10.5 with docker compose but still had get this problem neither rebuilding all containers nor changing to hostname_callable to airflow.utils.net.get_host_ip_address.

At this time (https://github.com/apache/airflow/issues/42136#issuecomment-2357283522), I used the airflow db upgrade command , but I realized it has been deprecated. I retried the upgrade using the airflow db migrate -n "2.10.2" command, and it works for me now.

https://airflow.apache.org/docs/apache-airflow/2.10.0/installation/upgrading.html#offline-sql-migration-scripts

I’m having this issue using 3.0.0rc2, too. Thoughts? I’m using docker.

I use docker compose to run airflow and had the same problem for all dags. What I did was stopping all dags, mark all task which were up for retry as failed, and rebuild all containers related to airflow. Now I can run my dags.

try to check that dags exist in worker, schedule and webserver. I deploy Airflow in K8S and get this error when putting my dags into scheduler(expecting that in will replicate into another pods), but when I check dags folder in worker it was empty