airflow: TypeError: cannot pickle 'module' object

Apache Airflow version

2.5.0

What happened

I’m running airflow in Kubernetes, I schedule PythonOperator using Kubernetes executor to download a dataset from s3 and I’m getting this error:

[2023-01-12 07:42:14,502] {dagbag.py:538} INFO - Filling up the DagBag from /opt/airflow/dags/recognizer_pipeline.py
[2023-01-12 07:42:18,298] {font_manager.py:1633} INFO - generated new fontManager
Downloading https://ultralytics.com/assets/Arial.ttf to /home/airflow/.config/Ultralytics/Arial.ttf...
[2023-01-12 07:42:19,779] {s3.py:48} INFO - Initializing S3 client
[2023-01-12 07:42:19,800] {credentials.py:1108} INFO - Found credentials from IAM Role: high_load-eks-node-group-20220721173708602000000002
[2023-01-12 07:42:19,904] {s3.py:48} INFO - Initializing S3 client
[2023-01-12 07:42:21,855] {task_command.py:389} INFO - Running <TaskInstance: recognizer-pipeline.download_dataset.download_raw_weights manual__2023-01-12T07:41:48.392745+00:00 [queued]> on host recognizer-pipeline-download-d-3765e820b7594e71988db6c28c48ae66
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 39, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 108, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 396, in task_run
    _run_task_by_selected_method(args, dag, ti)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 193, in _run_task_by_selected_method
    _run_task_by_local_task_job(args, ti)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 252, in _run_task_by_local_task_job
    run_job.run()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 247, in run
    self._execute()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 132, in _execute
    self.handle_task_exit(return_code)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 164, in handle_task_exit
    self.task_instance.schedule_downstream_tasks()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2562, in schedule_downstream_tasks
    partial_dag = task.dag.partial_subset(
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2201, in partial_subset
    dag.task_dict = {
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2202, in <dictcomp>
    t.task_id: _deepcopy_task(t)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dag.py", line 2199, in _deepcopy_task
    return copy.deepcopy(t, memo)
  File "/usr/local/lib/python3.8/copy.py", line 153, in deepcopy
    y = copier(memo)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1166, in __deepcopy__
    setattr(result, k, copy.deepcopy(v, memo))
  File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.8/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.8/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.8/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.8/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'module' object

and when I check the logs of the task in the web server I find this:

*** Reading local file: /opt/airflow/logs/dag_id=recognizer-pipeline/run_id=manual__2023-01-12T07:41:48.392745+00:00/task_id=download_dataset.download_platesmania_test_dataset/attempt=1.log
[2023-01-12, 07:42:19 UTC] {taskinstance.py:1087} INFO - Dependencies all met for <TaskInstance: recognizer-pipeline.download_dataset.download_platesmania_test_dataset manual__2023-01-12T07:41:48.392745+00:00 [queued]>
[2023-01-12, 07:42:19 UTC] {taskinstance.py:1087} INFO - Dependencies all met for <TaskInstance: recognizer-pipeline.download_dataset.download_platesmania_test_dataset manual__2023-01-12T07:41:48.392745+00:00 [queued]>
[2023-01-12, 07:42:19 UTC] {taskinstance.py:1283} INFO - 
--------------------------------------------------------------------------------
[2023-01-12, 07:42:19 UTC] {taskinstance.py:1284} INFO - Starting attempt 1 of 2
[2023-01-12, 07:42:19 UTC] {taskinstance.py:1285} INFO - 
--------------------------------------------------------------------------------
[2023-01-12, 07:42:19 UTC] {taskinstance.py:1304} INFO - Executing <Task(PythonOperator): download_dataset.download_platesmania_test_dataset> on 2023-01-12 07:41:48.392745+00:00
[2023-01-12, 07:42:19 UTC] {standard_task_runner.py:55} INFO - Started process 26 to run task
[2023-01-12, 07:42:19 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'recognizer-pipeline', 'download_dataset.download_platesmania_test_dataset', 'manual__2023-01-12T07:41:48.392745+00:00', '--job-id', '57', '--raw', '--subdir', 'DAGS_FOLDER/recognizer_pipeline.py', '--cfg-path', '/tmp/tmp4pmu878r']
[2023-01-12, 07:42:19 UTC] {standard_task_runner.py:83} INFO - Job 57: Subtask download_dataset.download_platesmania_test_dataset
[2023-01-12, 07:42:19 UTC] {task_command.py:389} INFO - Running <TaskInstance: recognizer-pipeline.download_dataset.download_platesmania_test_dataset manual__2023-01-12T07:41:48.392745+00:00 [running]> on host recognizer-pipeline-download-d-ec7a86c95d6c442ca1e719f26129dd7e
[2023-01-12, 07:42:20 UTC] {pod_generator.py:424} WARNING - Model file  does not exist
[2023-01-12, 07:42:20 UTC] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1378, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1504, in _execute_task_with_callbacks
    airflow_context_vars = context_to_airflow_vars(context, in_env_var_format=True)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/operator_helpers.py", line 119, in context_to_airflow_vars
    params[mapping_value] = ",".join(_attr)
TypeError: sequence item 0: expected str instance, NoneType found
[2023-01-12, 07:42:20 UTC] {taskinstance.py:1322} INFO - Marking task as UP_FOR_RETRY. dag_id=recognizer-pipeline, task_id=download_dataset.download_platesmania_test_dataset, execution_date=20230112T074148, start_date=20230112T074219, end_date=20230112T074220
[2023-01-12, 07:42:20 UTC] {standard_task_runner.py:100} ERROR - Failed to execute job 57 for task download_dataset.download_platesmania_test_dataset (sequence item 0: expected str instance, NoneType found; 26)
[2023-01-12, 07:42:20 UTC] {local_task_job.py:159} INFO - Task exited with return code 1

What you think should happen instead

The correct behavior is that the task runs as a Kubernetes pod and downloads the data to a specific volume.

How to reproduce

Airflow scheduler and webserver:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: airflow
  namespace: airflow
spec:
  replicas: 1
  selector:
    matchLabels:
      app: airflow
  template:
    metadata:
      labels:
        app: airflow
      name: airflow
    spec:
      serviceAccountName: airflow-scheduler
      containers:
        - name: airflow-scheduler
          image: ${TF_ECR_AIRFLOW}:latest
          imagePullPolicy: Always
          env:
            - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
              valueFrom:
                secretKeyRef:
                  name: airflow-staging-manual
                  key: SQLALCHEMY_DATABASE_URI
            - name: AIRFLOW__CORE__EXECUTOR
              value: KubernetesExecutor
            - name: AIRFLOW__CORE__ENABLE_XCOM_PICKLING
              value: "True"
            - name: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_REPOSITORY
              value: ${TF_ECR_AIRFLOW}
            - name: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG
              value: latest
            - name: ECR_AIRFLOW_GPU
              valueFrom:
                configMapKeyRef:
                  name: airflow-infrastructure
                  key: TF_ECR_AIRFLOW_GPU
            - name: AWS_ACCESS_KEY_ID_S3
              valueFrom:
                configMapKeyRef:
                  name: airflow-infrastructure-k8s
                  key: IAM_AIRFLOW_ID
            - name: AWS_SECRET_ACCESS_KEY_S3
              valueFrom:
                secretKeyRef:
                  name: airflow-staging-managed
                  key: IAM_SECRET
            - name: AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE
              value: airflow
            - name: AIRFLOW__KUBERNETES_EXECUTOR__POD_TEMPLATE_FILE
              value: "/opt/airflow/pod-creator.yml"
            - name: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS
              value: "True"
            - name: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE
              value: "True"
          volumeMounts:
            - name: airflow-logs
              mountPath: /opt/airflow/logs
          command:
            - airflow
          args:
            - scheduler
        - name: airflow-webserver
          env:
            - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
              valueFrom:
                secretKeyRef:
                  name: airflow-staging-manual
                  key: SQLALCHEMY_DATABASE_URI
          image: ${TF_ECR_AIRFLOW}:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          command:
            - airflow
          args:
            - webserver
          volumeMounts:
            - name: airflow-logs
              mountPath: /opt/airflow/logs
      restartPolicy: Always
      securityContext:
        runAsUser: 50000
        runAsGroup: 0
        fsGroup: 0
      volumes:
        - name: airflow-logs
          persistentVolumeClaim:
            claimName: airflow-logs

Pod creator template:

apiVersion: v1
kind: Pod
metadata:
  name: airflow-worker
  namespace: airflow
spec:
  containers: 
    - name: base 
      imagePullPolicy: Always
      image: 958824886822.dkr.ecr.eu-central-1.amazonaws.com/airflow:latest
      env: 
      - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
        valueFrom:
          secretKeyRef:
            name: airflow-staging-manual
            key: SQLALCHEMY_DATABASE_URI
      - name: AIRFLOW__CORE__EXECUTOR
        value: "LocalExecutor"
      - name: AIRFLOW__CORE__ENABLE_XCOM_PICKLING
        value: "True"
      - name: AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT
        value: 60.0
      - name: AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE
        value: "airflow"
      - name: AIRFLOW__CORE__DAGS_FOLDER
        value: "/opt/airflow/dags"
      volumeMounts:
        - name: airflow-logs
          mountPath: /opt/airflow/logs
        - name: ml-training-data
          mountPath: /opt/airflow/data
  restartPolicy: Never
  securityContext:
    runAsUser: 50000
    runAsGroup: 0
    fsGroup: 0
  serviceAccountName: "airflow-scheduler"
  volumes:
    - name: airflow-logs
      persistentVolumeClaim:
        claimName: airflow-logs
    - name: ml-training-data
      persistentVolumeClaim:
        claimName: ml-training-data
 

Example Dag:


with DAG(
    'detector-pipeline',
    default_args=default_args,
    description='End-to-end ML pipeline for detector',
    schedule_interval=timedelta(days=1),
    start_date=days_ago(1),
    tags=[],) as dag:
    # task 1
    with TaskGroup('download_dataset') as download_dataset:
        PythonOperator(task_id=f'download_raw_weights',
                       python_callable=downloader.download_raw_weights)

        PythonOperator(task_id=f'download_camera_train_dataset',
                       python_callable=downloader.download_folder, 
                       op_kwargs={'data_dir': config.CAMERA_TRAIN_DIR})

        PythonOperator(task_id=f'download_camera_test_dataset',
                       python_callable=downloader.download_folder, 
                       op_kwargs={'data_dir': config.CAMERA_TEST_DIR})
        
        PythonOperator(task_id=f'download_platesmania_train_dataset',
                       python_callable=downloader.download_folder, 
                       op_kwargs={'data_dir': config.PLATESMANIA_TRAIN_DIR})

        PythonOperator(task_id=f'download_platesmania_test_dataset',
                       python_callable=downloader.download_folder, 
                       op_kwargs={'data_dir': config.PLATESMANIA_TEST_DIR})
        
        PythonOperator(task_id=f'download_augmentation_dataset',
                       python_callable=downloader.download_folder, 
                       op_kwargs={'data_dir': config.AUGMENTATION_DIR})

        PythonOperator(task_id=f'download_problematic_dataset',
                       python_callable=downloader.download_folder, 
                       op_kwargs={'data_dir': config.PROBLEMATIC_PLATES_DIR})

Operating System

Debian GNU/Linux 11(bullseye)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==6.2.0
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==5.0.0
apache-airflow-providers-common-sql==1.3.1
apache-airflow-providers-docker==3.3.0
apache-airflow-providers-elasticsearch==4.3.1
apache-airflow-providers-ftp==3.2.0
apache-airflow-providers-google==8.6.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.2.0
apache-airflow-providers-http==4.1.0
apache-airflow-providers-imap==3.1.0
apache-airflow-providers-microsoft-azure==5.0.0
apache-airflow-providers-mysql==3.4.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-postgres==5.3.1
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==4.2.0
apache-airflow-providers-slack==7.1.0
apache-airflow-providers-sqlite==3.3.1
apache-airflow-providers-ssh==3.3.0

Deployment

Other

Deployment details

No response

Anything else

I faced the same issue with Airflow 2.3.2

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

[Edited] I supposed that callback function with arguments is not working. But it is now working for me when I define function parameters explicitly (without **). Similar issue here: https://github.com/apache/airflow/issues/20974

That’s because the Kubernetes executor relies on pickling to send a function to the pod.