Flexget: Repeating OperationalError: (sqlite3.OperationalError) database is locked
I’m getting these errors, while my box upgrades itself at 4 AM using unattended-upgrades
.
There are some related tickets: #703, #336 and Trac issue #3030 - finally commit 0c74a6b
2016-03-29 04:00 ERROR lazy_lookup nC_movie-HUN Unhandled error in lazy lookup plugin
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flexget/utils/lazy_dict.py", line 36, in __getitem__
func(self.store)
File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/metainfo/imdb_lookup.py", line 235, in lazy_loader
self.lookup(entry)
File "/usr/local/lib/python2.7/dist-packages/flexget/plugin.py", line 124, in wrapped_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flexget/utils/database.py", line 30, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/metainfo/imdb_lookup.py", line 356, in lookup
entry.update_using_map(self.field_map, movie)
File "/usr/local/lib/python2.7/dist-packages/flexget/entry.py", line 266, in update_using_map
v = value(source_item)
File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/metainfo/imdb_lookup.py", line 212, in <lambda>
'imdb_languages': lambda movie: [lang.language.name for lang in movie.languages],
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
return self.impl.get(instance_state(instance), dict_)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 583, in get
value = self.callable_(state, passive)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 532, in _load_for_state
return self._emit_lazyload(session, state, ident_key, passive)
File "<string>", line 1, in <lambda>
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 602, in _emit_lazyload
result = q.all()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
return list(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'SELECT imdb_movie_languages.movie_id AS imdb_movie_languages_movie_id, imdb_movie_languages.language_id AS imdb_movie_languages_language_id, imdb_movie_languages.prominence AS imdb_movie_languages_prominence \nFROM imdb_movie_languages \nWHERE ? = imdb_movie_languages.movie_id ORDER BY imdb_movie_languages.prominence'] [parameters: (1495,)]
2016-03-31 04:05 CRITICAL task_queue Database error while running a task. Attempting to recover.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flexget/task_queue.py", line 47, in run
self.current_task.execute()
File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 69, in wrapper
return func(self, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 579, in execute
self._execute()
File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 511, in _execute
last_hash = session.query(TaskConfigHash).filter(TaskConfigHash.task == self.name).first()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first
ret = list(self[0:1])
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
return list(res)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'SELECT feed_config_hash.name AS feed_config_hash_name, feed_config_hash.id AS feed_config_hash_id, feed_config_hash.hash AS feed_config_hash_hash \nFROM feed_config_hash \nWHERE feed_config_hash.name = ?\n LIMIT ? OFFSET ?'] [parameters: (u'nC_notify-Progs', 1, 0)]
2016-04-01 03:57 CRITICAL task_queue Database error while running a task. Attempting to recover.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flexget/task_queue.py", line 47, in run
self.current_task.execute()
File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 69, in wrapper
return func(self, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 579, in execute
self._execute()
File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 511, in _execute
last_hash = session.query(TaskConfigHash).filter(TaskConfigHash.task == self.name).first()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first
ret = list(self[0:1])
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__
return list(res)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'SELECT feed_config_hash.name AS feed_config_hash_name, feed_config_hash.id AS feed_config_hash_id, feed_config_hash.hash AS feed_config_hash_hash \nFROM feed_config_hash \nWHERE feed_config_hash.name = ?\n LIMIT ? OFFSET ?'] [parameters: (u'nC_notify-Progs', 1, 0)]
2016-04-02 12:43 CRITICAL task nC_notify-Progs BUG: Unhandled error in plugin remember_rejected: (sqlite3.OperationalError) database is locked [SQL: u'DELETE FROM remember_rejected_entry WHERE remember_rejected_entry.feed_id = ? AND remember_rejected_entry.expires < ?'] [parameters: (2109, '2016-04-02 12:43:28.181146')]
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 438, in __run_plugin
return method(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flexget/event.py", line 22, in __call__
return self.func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/filter/remember_rejected.py", line 94, in on_task_start
filter(RememberEntry.expires < datetime.now()).delete()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3048, in delete
delete_op.exec_()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1127, in exec_
self._do_exec()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1311, in _do_exec
mapper=self.mapper)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1034, in execute
bind, close_with_result=True).execute(clause, params or {})
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) database is locked [SQL: u'DELETE FROM remember_rejected_entry WHERE remember_rejected_entry.feed_id = ? AND remember_rejected_entry.expires < ?'] [parameters: (2109, '2016-04-02 12:43:28.181146')]
1000th issue 😃 💯
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 66 (48 by maintainers)
Commits related to this issue
- [fix] series: save entry state before metainfo phase to avoid lazy lookups during backlog (timeframe), fixes #1000 — committed to Flexget/Flexget by cvium 6 years ago
- series: save entry state before metainfo phase to avoid lazy lookups during backlog (timeframe) (#2215) * [fix] series: save entry state before metainfo phase to avoid lazy lookups during backlog (ti... — committed to Flexget/Flexget by cvium 6 years ago
@frankyw @zosky Good news and bad news.
Good news: I’ve located the issue. The problem is with lazy fields (eg. trakt_lookup, imdb_lookup) and
backlog
plugin, which is used internally by seriestimeframe
. More specifically, this line https://github.com/Flexget/Flexget/blob/develop/flexget/plugins/filter/series.py#L2131 forces an evaluation of all lazy fields, which runs in a separate db session, butseries
already has a lock on the db at this point, which is why it crashes.A quick fix is to add
object_session(episode).commit()
before this line or simply don’t use lookup plugins withtimeframe
.Bad news: It’s not an easy thing to fix. It’s a fundamental problem with how we’ve implemented lazy lookups. It could take a long while to fix this properly. Sorry.
disabled the cron job and restarted daemon with debug – will have a look in the morning