dvc: init: fails on 2nd run with NTFS on Linux (gitdb lib bug)

DVC Version: 0.35.7

lsb_release -a Distributor ID: Ubuntu Description: Ubuntu 16.04.5 LTS Release: 16.04 Codename: xenial

Steps to reproduce:

git init
dvc init
rm -rf .dvc
dvc init

Adding '.dvc/state' to '.dvc/.gitignore'.
Adding '.dvc/lock' to '.dvc/.gitignore'.
Adding '.dvc/config.local' to '.dvc/.gitignore'.
Adding '.dvc/updater' to '.dvc/.gitignore'.
Adding '.dvc/updater.lock' to '.dvc/.gitignore'.
Adding '.dvc/state-journal' to '.dvc/.gitignore'.
Adding '.dvc/state-wal' to '.dvc/.gitignore'.
Adding '.dvc/cache' to '.dvc/.gitignore'.
ERROR: unexpected error - [Errno 13] Permission denied: '/mnt/c/Users/villasv/Projects/dvc-talk/.git/objects/obj80fx1hnj' -> '/mnt/c/Users/villasv/Projects/dvc-talk/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 18 (17 by maintainers)

Commits related to this issue

loose: only rename if needed Our user was experiencing issue [1] when using a git repository on NTFS mount running on Linux. The current check checks if we are running on Windows, but it should real... — committed to efiop/gitdb by efiop 5 years ago
loose: rename only if needed Our user was experiencing issue [1] when using a git repository on NTFS mount running on Linux. The current check checks if we are running on Windows, but it should reall... — committed to efiop/gitdb by efiop 5 years ago
loose: rename only if needed Our user was experiencing issue [1] when using a git repository on NTFS mount running on Linux. The current check checks if we are running on Windows, but it should reall... — committed to efiop/gitdb by efiop 5 years ago

Most upvoted comments

For the record, gitpython-developers/gitdb#52 got merged, now waiting for the new gitdb release, so we could update our requirements.

efiop on Sep 29, 2019

pip install --force-reinstall git+https://github.com/efiop/gitdb
./repro.sh
+ rm -rf myrepo
+ mkdir myrepo
+ cd myrepo
+ git init
Initialized empty Git repository in /mnt/c/Users/villasv/Projects/myrepo/.git/
+ dvc init
Adding '.dvc/config.local' to '.dvc/.gitignore'.
Adding '.dvc/updater' to '.dvc/.gitignore'.
Adding '.dvc/state-journal' to '.dvc/.gitignore'.
Adding '.dvc/state-wal' to '.dvc/.gitignore'.
Adding '.dvc/state' to '.dvc/.gitignore'.
Adding '.dvc/lock' to '.dvc/.gitignore'.
Adding '.dvc/tmp' to '.dvc/.gitignore'.
Adding '.dvc/updater.lock' to '.dvc/.gitignore'.
Adding '.dvc/cache' to '.dvc/.gitignore'.

You can now commit the changes to git.

+---------------------------------------------------------------------+
|                                                                     |
|        DVC has enabled anonymous aggregate usage analytics.         |
|     Read the analytics documentation (and how to opt-out) here:     |
|              https://dvc.org/doc/user-guide/analytics               |
|                                                                     |
+---------------------------------------------------------------------+

What's next?
------------
- Check out the documentation: https://dvc.org/doc
- Get help and share ideas: https://dvc.org/chat
- Star us on GitHub: https://github.com/iterative/dvc
+ rm -rf .dvc
+ git reset
+ dvc init -v
Adding '.dvc/config.local' to '.dvc/.gitignore'.
Adding '.dvc/updater' to '.dvc/.gitignore'.
Adding '.dvc/state-journal' to '.dvc/.gitignore'.
Adding '.dvc/state-wal' to '.dvc/.gitignore'.
Adding '.dvc/state' to '.dvc/.gitignore'.
Adding '.dvc/lock' to '.dvc/.gitignore'.
Adding '.dvc/tmp' to '.dvc/.gitignore'.
Adding '.dvc/updater.lock' to '.dvc/.gitignore'.
Adding '.dvc/cache' to '.dvc/.gitignore'.

You can now commit the changes to git.

DEBUG: Analytics is enabled.
+---------------------------------------------------------------------+
|                                                                     |
|        DVC has enabled anonymous aggregate usage analytics.         |
|     Read the analytics documentation (and how to opt-out) here:     |
|              https://dvc.org/doc/user-guide/analytics               |
|                                                                     |
+---------------------------------------------------------------------+

What's next?
------------
- Check out the documentation: https://dvc.org/doc
- Get help and share ideas: https://dvc.org/chat
- Star us on GitHub: https://github.com/iterative/dvc

It seems it works as expected! 🎉

villasv on Sep 27, 2019

@jorgeorpinel NTFS and yes https://github.com/gitpython-developers/gitdb/pull/52

efiop on Sep 26, 2019

I dug into gitdb and this is the exact point of failure:

            # rename onto existing doesn't work on windows
            if os.name == 'nt':
                if isfile(obj_path):
                    remove(tmp_path)
                else:
                    rename(tmp_path, obj_path)
                # end rename only if needed
            else:
                rename(tmp_path, obj_path)
            # END handle win32

It seems they’re attempting to handle NTFS, but doing so with an OS check. This fails for me because I have NTFS but under Linux 😕

villasv on May 20, 2019

@villasv what happens if you run git add .dvc/config after this exception? Could you also run

ls -la for these last two git object (a link and an object):

'/mnt/c/Users/villasv/Projects/myrepo/.git/objects/obj4wr1vomg' -> '/mnt/c/Users/villasv/Projects/myrepo/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391

Do they have a regular user/group?

Is it possible that there are some security settings on Linux (this is Linux right) that prevent access to any other process to these files except git itself? What do you think, @efiop ?

shcheklein on Apr 16, 2019

I’m using Bash on WSL, so maybe something funny is happening at the filesystem level? I was able to reproduce (by accident) with an old version as well, dvc==0.9.5 I think, so it’s probably something with gitpython.

+ rm -rf myrepo
+ mkdir myrepo
+ cd myrepo
+ git init
Initialized empty Git repository in /mnt/c/Users/villasv/Projects/myrepo/.git/
+ dvc init
Adding '.dvc/state' to '.dvc/.gitignore'.
Adding '.dvc/lock' to '.dvc/.gitignore'.
Adding '.dvc/config.local' to '.dvc/.gitignore'.
Adding '.dvc/updater' to '.dvc/.gitignore'.
Adding '.dvc/updater.lock' to '.dvc/.gitignore'.
Adding '.dvc/state-journal' to '.dvc/.gitignore'.
Adding '.dvc/state-wal' to '.dvc/.gitignore'.
Adding '.dvc/cache' to '.dvc/.gitignore'.

You can now commit the changes to git.

+---------------------------------------------------------------------+
|                                                                     |
|        DVC has enabled anonymous aggregate usage analytics.         |
|     Read the analytics documentation (and how to opt-out) here:     |
|              https://dvc.org/doc/user-guide/analytics               |
|                                                                     |
+---------------------------------------------------------------------+

What's next?
------------
- Check out the documentation: https://dvc.org/doc
- Get help and share ideas: https://dvc.org/chat
- Star us on GitHub: https://github.com/iterative/dvc
+ rm -rf .dvc
+ dvc init -v
ERROR: you are not inside of a dvc repository (checked up to mount point '/mnt/c')

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!ERROR: you are not inside of a dvc repository (checked up to mount point '/mnt/c')

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!

Adding '.dvc/state' to '.dvc/.gitignore'.
Adding '.dvc/lock' to '.dvc/.gitignore'.
Adding '.dvc/config.local' to '.dvc/.gitignore'.
Adding '.dvc/updater' to '.dvc/.gitignore'.
Adding '.dvc/updater.lock' to '.dvc/.gitignore'.
Adding '.dvc/state-journal' to '.dvc/.gitignore'.
Adding '.dvc/state-wal' to '.dvc/.gitignore'.
Adding '.dvc/cache' to '.dvc/.gitignore'.
DEBUG: Trying to spawn '['/usr/bin/python3', '-m', 'dvc', 'daemon', '-q', 'updater']'
DEBUG: Spawned '['/usr/bin/python3', '-m', 'dvc', 'daemon', '-q', 'updater']'
ERROR: unexpected error - [Errno 13] Permission denied: '/mnt/c/Users/villasv/Projects/myrepo/.git/objects/obj4wr1vomg' -> '/mnt/c/Users/villasv/Projects/myrepo/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/dvc/main.py", line 37, in main
    ret = cmd.run_cmd()
  File "/usr/local/lib/python3.5/dist-packages/dvc/command/init.py", line 22, in run_cmd
    ".", no_scm=self.args.no_scm, force=self.args.force
  File "/usr/local/lib/python3.5/dist-packages/dvc/repo/__init__.py", line 116, in init
    init(root_dir=root_dir, no_scm=no_scm, force=force)
  File "/usr/local/lib/python3.5/dist-packages/dvc/repo/init.py", line 88, in init
    scm.add([config.config_file])
  File "/usr/local/lib/python3.5/dist-packages/dvc/scm/git/__init__.py", line 158, in add
    self.git.index.add(paths)
  File "/usr/local/lib/python3.5/dist-packages/git/index/base.py", line 742, in add
    entries_added.extend(self._entries_for_paths(paths, path_rewriter, fprogress, entries))
  File "/usr/local/lib/python3.5/dist-packages/git/util.py", line 72, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/git/index/util.py", line 91, in set_git_working_dir
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/git/index/base.py", line 628, in _entries_for_paths
    entries_added.append(self._store_path(filepath, fprogress))
  File "/usr/local/lib/python3.5/dist-packages/git/index/base.py", line 597, in _store_path
    istream = self.repo.odb.store(IStream(Blob.type, st.st_size, stream))
  File "/usr/local/lib/python3.5/dist-packages/gitdb/db/loose.py", line 236, in store
    rename(tmp_path, obj_path)
PermissionError: [Errno 13] Permission denied: '/mnt/c/Users/villasv/Projects/myrepo/.git/objects/obj4wr1vomg' -> '/mnt/c/Users/villasv/Projects/myrepo/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391'
------------------------------------------------------------

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
DEBUG: Analytics is enabled.
DEBUG: Trying to spawn '['/usr/bin/python3', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmpq0f9nw8i']'
DEBUG: Spawned '['/usr/bin/python3', '-m', 'dvc', 'daemon', '-q', 'analytics', '/tmp/tmpq0f9nw8i']'

villasv on Apr 15, 2019