nbformat: sqlite3.OperationalError: database is locked
I’ve deployed a JupyterHub instance and I’m running into a sqlite3.OperationalError: database is locked
from nbformat/sign.py
whenever I try to open a notebook. I can open the user/samlau95/tree
URL, but clicking a notebook or trying to create a new notebook hangs for ~45 seconds until it fails with a 504 Gateway error.
Later, the container running the notebook server will output:
[E 2016-10-05 19:44:18.016 samlau95 handlers:468] Unhandled error in API request
Traceback (most recent call last):
File "/opt/conda/lib/python3.5/site-packages/traitlets/traitlets.py", line 501, in get
value = obj._trait_values[self.name]
KeyError: 'db'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.5/site-packages/notebook/base/handlers.py", line 457, in wrapper
result = yield gen.maybe_future(method(self, *args, **kwargs))
File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
value = future.result()
File "/opt/conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
raise_exc_info(self._exc_info)
File "<string>", line 3, in raise_exc_info
File "/opt/conda/lib/python3.5/site-packages/tornado/gen.py", line 282, in wrapper
yielded = next(result)
File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/handlers.py", line 124, in get
path=path, type=type, format=format, content=content,
File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/filemanager.py", line 358, in get
model = self._notebook_model(path, content=content)
File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/filemanager.py", line 318, in _notebook_model
self.mark_trusted_cells(nb, path)
File "/opt/conda/lib/python3.5/site-packages/notebook/services/contents/manager.py", line 447, in mark_trusted_cells
trusted = self.notary.check_signature(nb)
File "/opt/conda/lib/python3.5/site-packages/nbformat/sign.py", line 220, in check_signature
if self.db is None:
File "/opt/conda/lib/python3.5/site-packages/traitlets/traitlets.py", line 529, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.5/site-packages/traitlets/traitlets.py", line 508, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.5/site-packages/nbformat/sign.py", line 127, in _db_default
self.init_db(db)
File "/opt/conda/lib/python3.5/site-packages/nbformat/sign.py", line 139, in init_db
)""")
sqlite3.OperationalError: database is locked
I can verify that the database is locked:
$ sqlite3 .local/share/jupyter/nbsignatures.db
SQLite version 3.11.0 2016-02-15 17:29:24
Enter ".help" for usage hints.
sqlite> .schema
Error: database is locked
And that the process is the notebook server:
$ fuser .local/share/jupyter/nbsignatures.db
/home/samlau95/.local/share/jupyter/nbsignatures.db: 23697
$ ps aux | grep 23697
root@sam-node1:/home/samlau95# ps aux | grep 23697
samlau95 23697 0.0 0.6 188472 48464 ? S 04:26 0:01 python3 /usr/local/bin/jupyterhub-singleuser --port=8888 --ip=0.0.0.0 --user=samlau95 --cookie-name=jupyter-hub-token-samlau95 --base-url=/user/samlau95 --hub-prefix=/hub/ --hub-api-url=http://10.128.0.7:8081/hub/api
This is running on Ubuntu 16.04 using the setup in https://github.com/data-8/jupyterhub-deploy which has been successfully deployed multiple times. This is the first time I’m deploying this on Ubuntu 16.04 (we’ve used 14.04 before) so perhaps this is related?
Here are the versions of packages installed:
root@4f4030e758f9:~# pip show notebook
---
Metadata-Version: 1.1
Name: notebook
Version: 4.2.1
Summary: A web-based notebook environment for interactive computing
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.com
License: BSD
Location: /opt/conda/lib/python3.5/site-packages
Requires:
Classifiers:
Intended Audience :: Developers
Intended Audience :: System Administrators
Intended Audience :: Science/Research
License :: OSI Approved :: BSD License
Programming Language :: Python
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
root@4f4030e758f9:~# pip show nbformat
---
Metadata-Version: 1.1
Name: nbformat
Version: 4.0.1
Summary: The Jupyter Notebook format
Home-page: http://jupyter.org
Author: Jupyter Development Team
Author-email: jupyter@googlegroups.com
License: BSD
Location: /opt/conda/lib/python3.5/site-packages
Requires:
Classifiers:
Intended Audience :: Developers
Intended Audience :: System Administrators
Intended Audience :: Science/Research
License :: OSI Approved :: BSD License
Programming Language :: Python
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.3
Any pointers on why this might be breaking? Happy to give more info.
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 16 (4 by maintainers)
I now have lines that look like
in my JupyterHub config but I’m still getting the same error in the logs. This is pretty puzzling to me since it seems like the issue is happening on db initialization. Any pointers?
The SQLite database should not be used on NFS. SQLite uses reader/writer locks to control access to the database. This locking mechanism might not work correctly if the database file is kept on an NFS filesystem. This is because fcntl() file locking is broken on many NFS implementations. Therefore, you should avoid putting SQLite database files on NFS since it will not handle well multiple processes which might try to access the file at the same time.
so ideally we should use PostgreSQL for production.
But can anyone help me how to change backend database in configuration for jupyterhub?
Another option is to clear the notebook output: https://gist.github.com/damianavila/5305869 That worked for me.
@takluyver Can you elaborate on how to do this please?