pyfilesystem2: Bad namespacing, imports breaks setuptools

fs.dropboxfs is implemented using fs as a namespace package. This causes it to misbehave when importing from fs for several reasons:

fs declares itself as a namespace package. According to the setuptools documentation here,

A) only namespace packages are supposed to contain the __import__('pkg_resources')... B) they must not contain executable code aside from that import.

We violate both of those rules, thus resulting in weird exceptions like this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dargueta/dev/goodrx-data-scripts/airflow_dags/grx/third_party/dropbox.py", line 6, in <module>
    from fs.dropboxfs import DropboxFS
  File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/dropboxfs/__init__.py", line 1, in <module>
    from .dropboxfs import DropboxFile
  File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/dropboxfs/dropboxfs.py", line 12, in <module>
    from fs.base import FS
  File "/Users/dargueta/.pyenv/versions/2.7.15/envs/gds/lib/python2.7/site-packages/fs/base.py", line 24, in <module>
    from . import copy, errors, fsencode, iotools, move, tools, walk, wildcard
ImportError: cannot import name fsencode

cannot import name fsencode

Changing that line to these two fixes the problem:

from . import copy, errors, iotools, move, tools, walk, wildcard
from ._fscompat import fsencode

There’s a similar issue in fs.test:

# Broken
from fs import ResourceType, Seek

# Works
from fs.enums import ResourceType, Seek

It appears these are the only two places in the repo we do this. Fixing these imports gets us a separate weird error in code using fs.dropboxfs but that may be unrelated.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 23 (17 by maintainers)

Most upvoted comments

Really sorry about wasting everyone’s time.

Not at all! We all learned something.

willmcgugan on Aug 14, 2019

I recommend against publishing filesystems in the fs namespace. I can see the attraction of namespace packages, there is an appealing elegance about it. But in practice it’s a real pain to work with. Particularly in development. Namespace packages don’t play well with installing packages in editable mode for instance.

So the official recommendations don’t use namespace packages. The only bit of magic required is the entry_points argument in setup.py which makes the filesystem available as an opener. In all other respects, you can publish a filesystem as any other Python module.

That said, there are several fs. namespace packages on PyPi and AFAIK they work just fine. So I’d be surprised if there was some fundamental breakage there.

I’ll do a deep dive in to namespace packages at some point, so I can properly evaluate @dargueta 's PR. In the meantime, our resident namespaces expert is @althonos If he’s not too busy, he might be able to shed some light on this.

willmcgugan on Aug 14, 2019