nbformat: sqlite3.OperationalError: database is locked

I’ve deployed a JupyterHub instance and I’m running into a sqlite3.OperationalError: database is locked from nbformat/sign.py whenever I try to open a notebook. I can open the user/samlau95/tree URL, but clicking a notebook or trying to create a new notebook hangs for ~45 seconds until it fails with a 504 Gateway error.

Later, the container running the notebook server will output:

[E 2016-10-05 19:44:18.016 samlau95 handlers:468] Unhandled error in API request
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/traitlets/traitlets.py", line 501, in get
        value = obj._trait_values[self.name]
    KeyError: 'db'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/notebook/base/handlers.py", line 457, in wrapper
        result = yield gen.maybe_future(method(self, *args, **kwargs))
      File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
        value = future.result()
      File "/opt/conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
      File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 282, in wrapper
        yielded = next(result)
      File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/handlers.py", line 124, in get
        path=path, type=type, format=format, content=content,
      File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/filemanager.py", line 358, in get
        model = self._notebook_model(path, content=content)
      File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/filemanager.py", line 318, in _notebook_model
        self.mark_trusted_cells(nb, path)
      File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/manager.py", line 447, in mark_trusted_cells
        trusted = self.notary.check_signature(nb)
      File "/opt/conda/lib/python3.5/site-packages/nbformat/sign.py", line 220, in check_signature
        if self.db is None:
      File "/opt/conda/lib/python3.5/site-packages/traitlets/traitlets.py", line 529, in __get__
        return self.get(obj, cls)
      File "/opt/conda/lib/python3.5/site-packages/traitlets/traitlets.py", line 508, in get
        value = self._validate(obj, dynamic_default())
      File "/opt/conda/lib/python3.5/site-packages/nbformat/sign.py", line 127, in _db_default
        self.init_db(db)
      File "/opt/conda/lib/python3.5/site-packages/nbformat/sign.py", line 139, in init_db
        )""")
    sqlite3.OperationalError: database is locked

I can verify that the database is locked:

$ sqlite3 .local/share/jupyter/nbsignatures.db
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> .schema
Error: database is locked

And that the process is the notebook server:

$ fuser .local/share/jupyter/nbsignatures.db
/home/samlau95/.local/share/jupyter/nbsignatures.db: 23697
$ ps aux | grep 23697
root@sam-node1:/home/samlau95# ps aux | grep 23697
samlau95 23697  0.0  0.6 188472 48464 ?        S    04:26   0:01 python3 /usr/local/bin/jupyterhub-singleuser --port=8888 --ip=0.0.0.0 --user=samlau95 --cookie-name=jupyter-hub-token-samlau95 --base-url=/user/samlau95 --hub-prefix=/hub/ --hub-api-url=http://10.128.0.7:8081/hub/api

This is running on Ubuntu 16.04 using the setup in https://github.com/data-8/jupyterhub-deploy which has been successfully deployed multiple times. This is the first time I’m deploying this on Ubuntu 16.04 (we’ve used 14.04 before) so perhaps this is related?

Here are the versions of packages installed:

root@4f4030e758f9:~# pip show notebook

---
Metadata-Version: 1.1
Name: notebook
Version: 4.2.1
Summary: A web-based notebook environment for interactive computing
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.com
License: BSD
Location: /opt/conda/lib/python3.5/site-packages
Requires:
Classifiers:
  Intended Audience :: Developers
  Intended Audience :: System Administrators
  Intended Audience :: Science/Research
  License :: OSI Approved :: BSD License
  Programming Language :: Python
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3

root@4f4030e758f9:~# pip show nbformat

---
Metadata-Version: 1.1
Name: nbformat
Version: 4.0.1
Summary: The Jupyter Notebook format
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.com
License: BSD
Location: /opt/conda/lib/python3.5/site-packages
Requires:
Classifiers:
  Intended Audience :: Developers
  Intended Audience :: System Administrators
  Intended Audience :: Science/Research
  License :: OSI Approved :: BSD License
  Programming Language :: Python
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3
  Programming Language :: Python :: 3.3

Any pointers on why this might be breaking? Happy to give more info.

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 16 (4 by maintainers)

Most upvoted comments

I now have lines that look like

c = get_config()
c.NotebookNotary.db_file = ':memory:'

in my JupyterHub config but I’m still getting the same error in the logs. This is pretty puzzling to me since it seems like the issue is happening on db initialization. Any pointers?

The SQLite database should not be used on NFS. SQLite uses reader/writer locks to control access to the database. This locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. Therefore, you should avoid putting SQLite database files on NFS since it will not handle well multiple processes which might try to access the file at the same time.

so ideally we should use PostgreSQL for production.

But can anyone help me how to change backend database in configuration for jupyterhub?

Another option is to clear the notebook output: https://gist.github.com/damianavila/5305869 That worked for me.

That needs to be configured for the individual notebook servers, not the hub.

@takluyver Can you elaborate on how to do this please?