Flexget: Repeating OperationalError: (sqlite3.OperationalError) database is locked

I’m getting these errors, while my box upgrades itself at 4 AM using unattended-upgrades. There are some related tickets: #703, #336 and Trac issue #3030 - finally commit 0c74a6b

2016-03-29 04:00 ERROR    lazy_lookup   nC_movie-HUN    Unhandled error in lazy lookup plugin
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flexget/utils/lazy_dict.py", line 36, in __getitem__
    func(self.store)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/metainfo/imdb_lookup.py", line 235, in lazy_loader
    self.lookup(entry)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugin.py", line 124, in wrapped_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flexget/utils/database.py", line 30, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/metainfo/imdb_lookup.py", line 356, in lookup
    entry.update_using_map(self.field_map, movie)
  File "/usr/local/lib/python2.7/dist-packages/flexget/entry.py", line 266, in update_using_map
    v = value(source_item)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/metainfo/imdb_lookup.py", line 212, in <lambda>
    'imdb_languages': lambda movie: [lang.language.name for lang in movie.languages],
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 583, in get
    value = self.callable_(state, passive)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 532, in _load_for_state
    return self._emit_lazyload(session, state, ident_key, passive)
  File "<string>", line 1, in <lambda>
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 602, in _emit_lazyload
    result = q.all()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
    return list(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'SELECT imdb_movie_languages.movie_id AS imdb_movie_languages_movie_id, imdb_movie_languages.language_id AS imdb_movie_languages_language_id, imdb_movie_languages.prominence AS imdb_movie_languages_prominence \nFROM imdb_movie_languages \nWHERE ? = imdb_movie_languages.movie_id ORDER BY imdb_movie_languages.prominence'] [parameters: (1495,)]
2016-03-31 04:05 CRITICAL task_queue                    Database error while running a task. Attempting to recover.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flexget/task_queue.py", line 47, in run
    self.current_task.execute()
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 69, in wrapper
    return func(self, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 579, in execute
    self._execute()
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 511, in _execute
    last_hash = session.query(TaskConfigHash).filter(TaskConfigHash.task == self.name).first()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first
    ret = list(self[0:1])
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
    return list(res)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'SELECT feed_config_hash.name AS feed_config_hash_name, feed_config_hash.id AS feed_config_hash_id, feed_config_hash.hash AS feed_config_hash_hash \nFROM feed_config_hash \nWHERE feed_config_hash.name = ?\n LIMIT ? OFFSET ?'] [parameters: (u'nC_notify-Progs', 1, 0)]
2016-04-01 03:57 CRITICAL task_queue                    Database error while running a task. Attempting to recover.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flexget/task_queue.py", line 47, in run
    self.current_task.execute()
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 69, in wrapper
    return func(self, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 579, in execute
    self._execute()
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 511, in _execute
    last_hash = session.query(TaskConfigHash).filter(TaskConfigHash.task == self.name).first()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first
    ret = list(self[0:1])
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
    return list(res)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
    return self._execute_and_instances(context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'SELECT feed_config_hash.name AS feed_config_hash_name, feed_config_hash.id AS feed_config_hash_id, feed_config_hash.hash AS feed_config_hash_hash \nFROM feed_config_hash \nWHERE feed_config_hash.name = ?\n LIMIT ? OFFSET ?'] [parameters: (u'nC_notify-Progs', 1, 0)]
2016-04-02 12:43 CRITICAL task          nC_notify-Progs BUG: Unhandled error in plugin remember_rejected: (sqlite3.OperationalError) database is locked [SQL: u'DELETE FROM remember_rejected_entry WHERE remember_rejected_entry.feed_id = ? AND remember_rejected_entry.expires < ?'] [parameters: (2109, '2016-04-02 12:43:28.181146')]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 438, in __run_plugin
    return method(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flexget/event.py", line 22, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/filter/remember_rejected.py", line 94, in on_task_start
    filter(RememberEntry.expires < datetime.now()).delete()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3048, in delete
    delete_op.exec_()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1127, in exec_
    self._do_exec()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1311, in _do_exec
    mapper=self.mapper)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1034, in execute
    bind, close_with_result=True).execute(clause, params or {})
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'DELETE FROM remember_rejected_entry WHERE remember_rejected_entry.feed_id = ? AND remember_rejected_entry.expires < ?'] [parameters: (2109, '2016-04-02 12:43:28.181146')]

1000th issue 😃 💯

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 66 (48 by maintainers)

Commits related to this issue

Most upvoted comments

@frankyw @zosky Good news and bad news.

Good news: I’ve located the issue. The problem is with lazy fields (eg. trakt_lookup, imdb_lookup) and backlog plugin, which is used internally by series timeframe. More specifically, this line https://github.com/Flexget/Flexget/blob/develop/flexget/plugins/filter/series.py#L2131 forces an evaluation of all lazy fields, which runs in a separate db session, but series already has a lock on the db at this point, which is why it crashes.

A quick fix is to add object_session(episode).commit() before this line or simply don’t use lookup plugins with timeframe.

Bad news: It’s not an easy thing to fix. It’s a fundamental problem with how we’ve implemented lazy lookups. It could take a long while to fix this properly. Sorry.

disabled the cron job and restarted daemon with debug – will have a look in the morning