airflow: airflow 2.0.0 sqlite3.OperationalError syntax error

Why I keep got this error when scheduler run my task?

[2020-12-31 03:05:22,716] {taskinstance.py:1396} ERROR - (sqlite3.OperationalError) near ",": syntax error
[SQL: DELETE FROM rendered_task_instance_fields WHERE rendered_task_instance_fields.dag_id = ? AND rendered_task_instance_fields.task_id = ? AND (rendered_task_instance_fields.dag_id, rendered_task_instance_fields.task_id, rendered_task_instance_fields.execution_date) NOT IN (SELECT rendered_task_instance_fields.dag_id, rendered_task_instance_fields.task_id, rendered_task_instance_fields.execution_date 
FROM rendered_task_instance_fields 
WHERE rendered_task_instance_fields.dag_id = ? AND rendered_task_instance_fields.task_id = ? ORDER BY rendered_task_instance_fields.execution_date DESC
 LIMIT ? OFFSET ?)]
[parameters: ('comic_app_v1', 'superman_task', 'comic_app_v1', 'superman_task', 30, 0)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Traceback (most recent call last):
  File "/home/user/py37/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
    cursor, statement, parameters, context
  File "/home/user/py37/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: near ",": syntax error

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 23 (13 by maintainers)

Commits related to this issue

Set minimum SQLite version supported. Some users reported that some older versions of SQLite do not work with Airflow 2.0. This happens for example with latest sqlite available by default on RHEL7 (s... — committed to PolideaInternal/airflow by potiuk 4 years ago
Set minimum SQLite version supported. (#13412) * Set minimum SQLite version supported. Some users reported that some older versions of SQLite do not work with Airflow 2.0. This happens for exampl... — committed to apache/airflow by potiuk 4 years ago
Check for minimum version of Sqlite Some users testing Airlfow 2.0 with sqlite noticed that for old versions of sqlite, Airflow does not run tasks and fails with 'sqlite3.OperationalError: near ",": ... — committed to PolideaInternal/airflow by potiuk 3 years ago
Check for minimum version of Sqlite (#13496) Some users testing Airlfow 2.0 with sqlite noticed that for old versions of sqlite, Airflow does not run tasks and fails with 'sqlite3.OperationalError:... — committed to apache/airflow by potiuk 3 years ago
Set minimum SQLite version supported. (#13412) * Set minimum SQLite version supported. Some users reported that some older versions of SQLite do not work with Airflow 2.0. This happens for example w... — committed to apache/airflow by potiuk 4 years ago
Check for minimum version of Sqlite (#13496) Some users testing Airlfow 2.0 with sqlite noticed that for old versions of sqlite, Airflow does not run tasks and fails with 'sqlite3.OperationalError: n... — committed to apache/airflow by potiuk 3 years ago

Most upvoted comments

You have a very old sqlite3 version. Please upgrade to latest version.

potiuk on Dec 31, 2020

FYI. 3.15.0 is the minimum version that works. I am adding it to the docs and Airflow will fail hard if sqlite3 version is less than that (PR is coming for #13493)

potiuk on Jan 5, 2021

To add to what Jarek said, sqlite should only be for local dev / unit tests (on your machine) but otherwise you should use Postgres / MySQL even for your company’s dev environment

kaxil on Jan 20, 2021

BTW. I confirmed that 3.11.0 results in this error. I followed the URL I gave you and downgraded sqlite with LD_LIBRARY_PATH and could reproduce it. I am trying to figure out which is the minimum version supported, but you should definitely upgrade sqlite3 and make it avaliable to python (via LD_LIBRARY_PATH for example)

potiuk on Jan 5, 2021

I think this is still a problem with your OS. My sqlite3_version is mych higher. If you look here https://sqlite.org/changes.html your version 3.11.0 is from 2016.

I think this might help you to figure out how to upgrade it. https://unix.stackexchange.com/questions/434100/updating-the-sqlite-version-used-by-python-3-on-centos-7

Python 3.7.9 (default, Nov 15 2020, 16:32:29) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> sqlite3.sqlite_version
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'sqlite3' is not defined
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.31.1'
>>> sqlite3.version
'2.6.0'
>>> 
``

potiuk on Jan 5, 2021