dvc: "unexpected error - unable to open database" file after updating from 0.59.2 to 0.71.0

Please provide information about your setup DVC version: 0.71.0 Platform: Ubuntu 18.04 Method of installation: pip

I’ve used dvc for managinf model files. I have not updated dvc for a while and after doing so going from 0.59.2 to 0.71.0 I could not add any new files to the existing dvc cache.

Here is the verbose output of the dvc add command

dvc add Empty\ Document -v
/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/git/repo/base.py:129: UserWarning: The use of environment variables in paths is deprecated
for security reasons and may be removed in the future!!
  warnings.warn("The use of environment variables in paths is deprecated" +
ERROR: unexpected error - unable to open database file
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/dvc/main.py", line 49, in main
    ret = cmd.run()
  File "/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/dvc/command/add.py", line 25, in run
    fname=self.args.file,
  File "/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/dvc/repo/__init__.py", line 35, in wrapper
    with repo.lock, repo.state:
  File "/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/dvc/state.py", line 136, in __enter__
    self.load()
  File "/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/dvc/state.py", line 228, in load
    self.database = _connect_sqlite(self.state_file, {"nolock": 1})
  File "/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/dvc/state.py", line 495, in _connect_sqlite
    return sqlite3.connect(uri, uri=True)
sqlite3.OperationalError: unable to open database file

Downgrading dvc version to 0.59.2 makes the problem dissapear again.

It feels like some backwards incopatibility issue?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 21 (13 by maintainers)

Commits related to this issue

state: quote % in path Fixes #2885 — committed to efiop/dvc by efiop 5 years ago

Most upvoted comments

@mroutis I used pip install dvc --upgrade doing pip uninstall dvc && pip install dvc like in your example did not change the behaviour

@efiop I tried updating dvc and then removing .dvc/state but still experience the same behaviour.

Here are the outputs: For the working version

dvc version
/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/git/repo/base.py:129: UserWarning: The use of environment variables in paths is deprecated
for security reasons and may be removed in the future!!
  warnings.warn("The use of environment variables in paths is deprecated" +
DVC version: 0.59.2
Python version: 3.7.4
Platform: Linux-4.15.0-70-generic-x86_64-with-debian-buster-sid
Binary: False
Cache: reflink - False, hardlink - False, symlink - True
Filesystem type (cache directory): ('fuseblk', '/dev/sda2')
Filesystem type (workspace): ('ext4', '/dev/sdb2')

and for the latest version

dvc version
/home/mlburk/anaconda3/envs/tf-gpu/lib/python3.7/site-packages/git/repo/base.py:129: UserWarning: The use of environment variables in paths is deprecated
for security reasons and may be removed in the future!!
  warnings.warn("The use of environment variables in paths is deprecated" +
DVC version: 0.71.0
Python version: 3.7.4
Platform: Linux-4.15.0-70-generic-x86_64-with-debian-buster-sid
Binary: False
Package: pip
Cache: reflink - False, hardlink - False, symlink - True
Filesystem type (cache directory): ('fuseblk', '/dev/sda2')
Filesystem type (workspace): ('ext4', '/dev/sdb2')

Output of python -c 'import sqlite3; print(str(sqlite3.sqlite_version_info))' (3, 29, 0)

Perhaps there some other small example I could also try to reproduce?

sremm on Dec 3, 2019

I couldn’t reproduce it @sremm, @efiop 😞

pip install dvc==0.59.2
dvc init --no-scm
dvc run -o foo "echo foo > foo"
pip uninstall dvc && pip install dvc
rm foo
dvc checkout
dvc run -o bar "echo bar > bar"

Maybe your state file got corrupted somehow.

How did you perform the update?

ghost on Dec 3, 2019

@sremm Great news! So we’ve found the issue. Patch is coming soon, stay tuned! 🙂 Thanks for the feedback!

efiop on Dec 5, 2019

Running the snipped below

import sqlite3

uri = "file:///home/mlburk/Repositories/Repo%2520Base/.dvc/state?nolock=1"
sqlite3.connect(uri, uri=True)

results in no errors and the state file is created in /home/mlburk/Repositories/Repo%20Base/.dvc/

sremm on Dec 5, 2019

@efiop So 😃 the prevous output from Ubuntu

Note that in my previous comment I also had to change the actual repository name, I realised that special characters might also have an effect and that there is the “%20” in the folder name as seen below, everything else is just letters. (same name on widows for the actualy repository folder)

python test_for_dvc_bug.py 
file:///home/mlburk/Repositories/Repo%20Base/.dvc/state?nolock=1
Traceback (most recent call last):
  File "test_for_dvc_bug.py", line 7, in <module>
    sqlite3.connect(uri, uri=True)
sqlite3.OperationalError: unable to open database file

and the new test test_for_dvc_bug_2.py

import sqlite3
import os

sqlite3.connect(os.path.join(os.getcwd(), ".dvc", "state"))

So this runs without errors. I guess this is what you expected since the “_build_sqlite_uri” function is under suspect. The “%20” in the name does not seem to be an issue for the sqlite3.connect() call though.

sremm on Dec 4, 2019