ipython: Sqlite ProgrammingError exception when embedding shell in thread

when using the embedded ipython shell in scrapy and exit one gets following exception:

 ProgrammingError: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 139779595237120 and this is thread id 139779675154208

to reproduce the bug apply the ipython 0.11 compatibilty patch to scrapy 0.12.0.2528: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636719 then:

scrapy shell http://www.dmoz.org/Computers/Programming/Languages/Python/Books/
...
>>> exit

or use

from twisted.internet import reactor, threads

def start_shell():
  import IPython
  IPython.embed()

threads.deferToThread(start_shell)
reactor.callLater(.3, reactor.stop)
reactor.run()

full traceback:

Error in sys.exitfunc:
---------------------------------------------------------------------------
ProgrammingError                          Traceback (most recent call last)
/usr/lib/python2.7/atexit.pyc in _run_exitfuncs()
     22         func, targs, kargs = _exithandlers.pop()
     23         try:
---> 24             func(*targs, **kargs)
     25         except SystemExit:
     26             exc_info = sys.exc_info()

/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.pyc in atexit_operations(self)
   2549 
   2550         # Close the history session (this stores the end time and line count)

-> 2551         self.history_manager.end_session()
   2552 
   2553         # Clear all user namespaces to release all references cleanly.


/usr/lib/python2.7/dist-packages/IPython/core/history.pyc in end_session(self)
    156     def end_session(self):
    157         """Close the database session, filling in the end time and line count."""
--> 158         self.writeout_cache()
    159         with self.db:
    160             self.db.execute("""UPDATE sessions SET end=?, num_cmds=? WHERE

/usr/lib/python2.7/dist-packages/IPython/core/history.pyc in writeout_cache(self, conn)
    437         with self.db_input_cache_lock:
    438             try:
--> 439                 self._writeout_input_cache(conn)
    440             except sqlite3.IntegrityError:
    441                 self.new_session(conn)

/usr/lib/python2.7/dist-packages/IPython/core/history.pyc in _writeout_input_cache(self, conn)
    422             for line in self.db_input_cache:
    423                 conn.execute("INSERT INTO history VALUES (?, ?, ?, ?)",
--> 424                                 (self.session_number,)+line)
    425 
    426     def _writeout_output_cache(self, conn):

ProgrammingError: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 139967009502976 and this is thread id 139967089420064

About this issue

Original URL
State: open
Created 13 years ago
Reactions: 2
Comments: 24 (6 by maintainers)

Commits related to this issue

Fix for #680 (https://github.com/ipython/ipython/issues/680) — committed to vintasoftware/ipython-2.3.1 by fjsj 9 years ago
Fix for #680 (https://github.com/ipython/ipython/issues/680) — committed to vintasoftware/ipython-2.3.1 by fjsj 9 years ago
Fix for #680 (https://github.com/ipython/ipython/issues/680) — committed to vintasoftware/ipython-2.3.1 by fjsj 9 years ago
Use check_same_thread=False by default for history sqlite db (#13886) I had run into the following exception while trying to use IPython in a thread: ``` Traceback (most recent call last): File... — committed to ipython/ipython by Carreau a year ago

Most upvoted comments

As a workaround, you can reopen the connection before the application exits:

from IPython.core import ipapi
from sqlite3 import connect
hist = ipapi.get().history_manager
hist.db = connect(hist.hist_file, check_same_thread=False)

akvadrako on Sep 21, 2011

That last one works as long as nothing else in your app uses atexit. I have a more targeted, but still very hackish 😛, approach at https://gist.github.com/mikenerone/786ce75cf8d906ae4ad1e0b57933c23f. That’s a trio example (“Have you, uh, met trio?”), but you can just grab the IPythonAtExitContext context manager out of that and do this where you want to embed:

with IPythonAtExitContext():
    IPython.embed(...)  # Or via the InteractiveShellEmbed class

mikenerone on Oct 12, 2020

bump

karolyi on Feb 25, 2018

In mid-2017, @akvadrako’s suggestion still holds. It looks something more like this now:

import sqlite

from IPython.core import get_ipython

# Reconnect to prevent sqlite3 ProgrammingError
hist = get_ipython().history_manager
hist.db = sqlite3.connect(hist.hist_file, check_same_thread=False)

dfee on May 5, 2017