airflow: Scheduler crashes when unpausing some dags with: TypeError: '>' not supported between instances of 'NoneType' and 'int'

Apache Airflow version: 2.0.0

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.15 Environment:

  • Cloud provider or hardware configuration: GKE
  • OS (e.g. from /etc/os-release): Ubuntu 18.04

What happened: I just migrated from 1.10.14 to 2.0.0. When I turn on some random dags, the scheduler crashes with the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
    self._run_scheduler_loop()
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1377, in _run_scheduler_loop
    num_queued_tis = self._do_scheduling(session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1533, in _do_scheduling
    num_queued_tis = self._critical_section_execute_task_instances(session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1132, in _critical_section_execute_task_instances
    queued_tis = self._executable_task_instances_to_queued(max_tis, session=session)
  File "/usr/local/lib/python3.6/dist-packages/airflow/utils/session.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/airflow/jobs/scheduler_job.py", line 1034, in _executable_task_instances_to_queued
    if task_instance.pool_slots > open_slots:
TypeError: '>' not supported between instances of 'NoneType' and 'int'

What you expected to happen:

I expected those dags would have their tasks scheduled without problems.

How to reproduce it:

Can’t reproduce it yet. Still trying to figure out if this happens only with specific dags or not.

Anything else we need to know:

I couldn’t find in which context task_instance.pool_slots could be None

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (16 by maintainers)

Commits related to this issue

Most upvoted comments

Another workaround is to run the following query in your metadata db:

UPDATE task_instance SET pool_slots = 1 WHERE pool_slots IS NULL;
+3
kaxil on Feb 4, 2021

i’m glad i found this before opening another issue 😂

Apache Airflow version: 2.0.1

What happened:

>>> import requests
>>> from requests.auth import HTTPBasicAuth
>>> r = requests.get("https://localhost:8080/api/v1/dags/{my_dag_id}/dagRuns/scheduled__2020-04-13T00%3A00%3A00%2B00%3A00/taskInstances", auth=HTTPBasicAuth('username', 'password'))
>>> r.status_code
500
>>> print(r.text)
{
  "detail": "None is not of type 'integer'\n\nFailed validating 'type' in schema['allOf'][0]['properties']
['task_instances']['items']['properties']['pool_slots']:\n    {'type': 'integer'}\n\nOn instance['task_ins
tances'][0]['pool_slots']:\n    None",
  "status": 500,
  "title": "Response body does not conform to specification",
  "type": "https://airflow.apache.org/docs/2.0.1/stable-rest-api-ref.html#section/Errors/Unknown"
}

>>> print(r.json()["detail"])
None is not of type 'integer'

Failed validating 'type' in schema['allOf'][0]['properties']['task_instances']['items']['properties']['poo
l_slots']:
    {'type': 'integer'}

On instance['task_instances'][0]['pool_slots']:
    None
+1
zachliu on Feb 23, 2021
Read more comments on GitHub
← airflow: S3 Remote Logging not working
airflow: Scheduler crashes with psycopg2.errors.DeadlockDetected exception →