ArchiveBox: Support for network drives or filesystems that don't implement FSYNC
Describe the bug
The issue encountered occurs when attempting to initialize an archivebox directory on a mounted network drive. The issue is reproducible in multiple directory locations located on the NAS. Initializing an archivebox directory on the local drive of the machine succeed. Additionally, if a successfully initialized directory is copied to the NAS, future archivebox operations on that copied directory fail. The write of the Config file appears to fail based on the traceback.
I investigated the trace output and added print statements to the atomicwrites function in which the failure occurs at line 46 of atomicwrites init.py Additionally, I uncommented the print statement at line 41 of archivebox system.py for further debug info.
Steps to reproduce
mkdir archivebox; cd archivebox; archivebox init
Screenshots or log output
OUTPUT:
$ archivebox init
[i] [2020-08-20 19:55:28] ArchiveBox v0.4.21: archivebox init
/Volumes/Public/data/archivebox
[+] Initializing a new ArchiveBox collection in this folder...
/Volumes/Public/data/archivebox
------------------------------------------------------------------
[+] Building archive folder structure...
√ /Volumes/Public/data/archivebox/sources
√ /Volumes/Public/data/archivebox/archive
√ /Volumes/Public/data/archivebox/logs
Atomic Write: w /Volumes/Public/data/archivebox/ArchiveBox.conf 437 overwrite=True
fd is: 3
fcntl.F_FULLFSYNC is 51
fd is: 3
fcntl.F_FULLFSYNC is 51
Traceback (most recent call last):
File "/Users/user123/.pyenv/versions/3.8.5/bin/archivebox", line 8, in <module>
sys.exit(main())
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/cli/__init__.py", line 122, in main
run_subcommand(
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/cli/__init__.py", line 62, in run_subcommand
module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/cli/archivebox_init.py", line 33, in main
init(
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/util.py", line 111, in typechecked_function
return func(*args, **kwargs)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/main.py", line 294, in init
write_config_file({}, out_dir=out_dir)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/config/__init__.py", line 376, in write_config_file
atomic_write(config_path, CONFIG_HEADER)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/util.py", line 111, in typechecked_function
return func(*args, **kwargs)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/system.py", line 46, in atomic_write
f.write(contents)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 171, in _open
self.commit(f)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 204, in commit
replace_atomic(f.name, self._path)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 101, in replace_atomic
return _replace_atomic(src, dst)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 58, in _replace_atomic
_sync_directory(os.path.normpath(os.path.dirname(dst)))
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 52, in _sync_directory
_proper_fsync(fd)
File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 46, in _proper_fsync
fcntl.fcntl(fd, fcntl.F_FULLFSYNC)
FileNotFoundError: [Errno 2] No such file or directory
Here is the output of mount for the two disks:
SUCCESS DRIVE:
/dev/disk1s5 on /System/Volumes/Data (apfs, local, journaled, nobrowse)
FAIL DRIVE:
//;AUTH=No%20User%20Authent@Drobo-5N2-RYE._afpovertcp._tcp.local/Public on /Volumes/Public (afpfs, nodev, nosuid, mounted by user123)
Here are the file permissions of the FAIL directory
drwxrwxrwx 1 user123 staff 264 Aug 20 15:55 archivebox
Software versions
- OS: MacOS 10.15.6 (19G73)
- ArchiveBox version: ArchiveBox v0.4.21
- Python version: Python 3.8.5
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 16 (8 by maintainers)
That was the hint I needed, thanks! I may not be doing things in the ideal way with my smb share, but I’m using
file_mode=0777,dir_mode=0777
as options in myfstab
, and I needed to add thenoperm
option for it to be able to respond to chmod calls with success (even if they aren’t actually making changes). I’m open to suggestions to better manage permissions, but I don’t think I have many options over smb.@pirate Thanks for your help with the investigation.
I have one suggestion/question: Is it possible to improve the initialization routine to output more descriptive debug info when a non-compatible drive destination is attempted for use?