airflow: Worker always return Invalid auth token when setting up demo environment using docker compose

Apache Airflow version

3.0.0

If “Other Airflow 2 version” selected, which one?

No response

What happened?

I’ve installed Airflow 3.0.0 using the docker compose instructions as described here: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html

I followed all steps and after running docker compose up, I logged into the admin user interface to run the example_simplest_dag . The DAG run fails and the worker shows this error log:

2025-04-23 18:14:50.194787 [info ] Task execute_workload[02ddde1a-8165-459e-91dd-518f2dd56c5d] received [celery.worker.strategy] 2025-04-23 18:14:50.304948 [info ] [02ddde1a-8165-459e-91dd-518f2dd56c5d] Executing workload in Celery: token=‘eyJ***’ ti=TaskInstance(id=UUID(‘019663dc-130a-764a-ac1b-d956964ee946’), task_id=‘my_task’, dag_id=‘example_simplest_dag’, run_id=‘manual__2025-04-23T18:14:49.335659+00:00’, try_number=1, map_index=-1, pool_slots=1, queue=‘default’, priority_weight=1, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath(‘example_simplest_dag.py’) bundle_info=BundleInfo(name=‘example_dags’, version=None) log_path=‘dag_id=example_simplest_dag/run_id=manual__2025-04-23T18:14:49.335659+00:00/task_id=my_task/attempt=1.log’ type=‘ExecuteTask’ [airflow.providers.celery.executors.celery_executor_utils] 2025-04-23 18:14:50.694139 [error ] Task execute_workload[02ddde1a-8165-459e-91dd-518f2dd56c5d] raised unexpected: ServerResponseError(‘Invalid auth token: Signature verification failed’) [celery.app.trace]

What you think should happen instead?

The DAG should return the result without errors.

How to reproduce

  1. create a fresh virtual machine running ubuntu server and install docker
  2. in an empty directory with a user allowed to run the dockercommands run the following steps:
  3. run curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.0.0/docker-compose.yaml'
  4. run mkdir -p ./dags ./logs ./plugins ./config
  5. run echo -e "AIRFLOW_UID=$(id -u)" > .env
  6. run `docker compose up airflow-init``
  7. run docker compose up
  8. if docker is not running on your machine, create ssh port forwarding to access the web-interface because otherwise you will run into this issue: #49622
  9. login with default credentials, navigate to the example_simplest_dagDAG and trigger the run

Operating System

Ubuntu 24.04.2 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-amazon | 9.6.1 apache-airflow-providers-celery | 3.10.6 apache-airflow-providers-cncf-kubernetes | 10.4.3 apache-airflow-providers-common-compat | 1.6.0 apache-airflow-providers-common-io | 1.5.4 apache-airflow-providers-common-messaging | 1.0.0 apache-airflow-providers-common-sql | 1.26.0 apache-airflow-providers-docker | 4.3.1 apache-airflow-providers-elasticsearch | 6.2.2 apache-airflow-providers-fab | 2.0.1 apache-airflow-providers-ftp | 3.12.3 apache-airflow-providers-git | 0.0.2 apache-airflow-providers-google | 15.1.0 apache-airflow-providers-grpc | 3.7.3 apache-airflow-providers-hashicorp | 4.1.1 apache-airflow-providers-http | 5.2.2 apache-airflow-providers-microsoft-azure | 12.3.1 apache-airflow-providers-mysql | 6.2.2 apache-airflow-providers-odbc | 4.9.2 apache-airflow-providers-openlineage | 2.2.0 apache-airflow-providers-postgres | 6.1.3 apache-airflow-providers-redis | 4.0.2 apache-airflow-providers-sendgrid | 4.0.1 apache-airflow-providers-sftp | 5.2.1 apache-airflow-providers-slack | 9.0.5 apache-airflow-providers-smtp | 2.0.3 apache-airflow-providers-snowflake | 6.2.2 apache-airflow-providers-ssh | 4.0.1 apache-airflow-providers-standard | 1.0.0

Deployment

Docker-Compose

Deployment details

docker compose version Docker Compose version v2.35.1

docker version Client: Docker Engine - Community Version: 28.1.1 API version: 1.49 Go version: go1.23.8 Git commit: 4eba377 Built: Fri Apr 18 09:52:14 2025 OS/Arch: linux/amd64 Context: default

Server: Docker Engine - Community Engine: Version: 28.1.1 API version: 1.49 (minimum version 1.24) Go version: go1.23.8 Git commit: 01f442b Built: Fri Apr 18 09:52:14 2025 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.27 GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da runc: Version: 1.2.5 GitCommit: v1.2.5-0-g59923ef docker-init: Version: 0.19.0 GitCommit: de40ad0

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 6
  • Comments: 16 (4 by maintainers)

Most upvoted comments

@inkerinmaa , @alonahmias Something like this will work Image

@inkerinmaa , yes all the containers would have its own airflow.cfg with different jwt_secret ( thats what I’m trying to fix) but the env variable override should force all containers to pick the one set in the env

@inkerinmaa , @alonahmias Something like this will work Image

@inkerinmaa , yes all the containers would have its own airflow.cfg with different jwt_secret ( thats what I’m trying to fix) but the env variable override should force all containers to pick the one set in the env

This worked for me.

For a workaround set export AIRFLOW__API_AUTH__JWT_SECRET=<some_random_string> you can generate one with openssl rand -base64 16

@amoghrajesh, @jscheffl can you assign this to me, I can provide fix

I followed the current proposed solution of this commit: https://github.com/apache/airflow/pull/49681/commits/37cf75962a5c013f7cf475cd942264e88205d9fe

By setting AIRFLOW_CONFIG: '/opt/airflow/config/airflow.cfg' in the docker-compose.yml I can get the DAG to run successfully and the worker to successfully submit the task execution result.

However, based on the discussion in #49681 around the potential implications of this configuration, I will wait for a long-term solution until marking this issue as resolved.

@inkerinmaa , @alonahmias Something like this will work Image

@inkerinmaa , yes all the containers would have its own airflow.cfg with different jwt_secret ( thats what I’m trying to fix) but the env variable override should force all containers to pick the one set in the env

I’ve tried this but no sign that it works. I still got error on airflow-worker

2025-06-17 01:05:16.469073 [info     ] [14503769-983a-4182-bec4-bf00e739d17b] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('01977b6b-4093-769f-bc02-366d653fc3ca'), task_id='check_status_survey', dag_id='dag-testing', run_id='scheduled__2025-06-17T01:04:00+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=2, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('testing/testing.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=dag-testing/run_id=scheduled__2025-06-17T01:04:00+00:00/task_id=check_status_survey/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-06-17 01:05:16.497594 [info     ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1
2025-06-17 01:05:17.108694 [warning  ] Server error                   [airflow.sdk.api.client] detail=None
2025-06-17 01:05:17.115715 [info     ] Process exited                 [supervisor] exit_code=-9 pid=59 signal_sent=SIGKILL
2025-06-17 01:05:17.123788 [error    ] Task execute_workload[14503769-983a-4182-bec4-bf00e739d17b] raised unexpected: ServerResponseError('Not Found') [celery.app.trace]

The docker-compose file screenshot is in the attachement. Besides all the environement variable, I only changed dash from the container name that the template to be using underscore.

Image

I face the same issue using the docker-compose setup on mac as well.

Image