pip: --target doesn't work for namespace packages

@Ivoz site.addsitedir('lib') was exactly what I needed. Thanks!

@JensTimmerman That’s an interesting idea. I agree, it’s a bit hostile to install packages into a --target and then they not be viable without some additional steps.

I’m not sure a warning is quite appropriate - as it may be the case that a tool invoking pip will be correcting for the condition as rwt does.

More importantly, though, this issue largely goes away with Python 3.3, so perhaps the best solution is just to work around it until you can port everything to Python 3.

So if I get this correct pip will not try to have a working namespace package when installed with --target? Could in this case some info (warning?) with some useful information be printed out to the user? e.g. pip doesn’t support this, you might want to run code for workaround with site package

In that latest thread, PJE says:

Setuptools has essentially always built non-egg, non-exe binary distributions in a PEP 420-compatible way (i.e., without __init__.py). And pkg_resources already builds a namespace path by asking importers if they can import a package at that location. So if PEP 420 importers say “yes” when asked to find_module(‘somepkg’) in the case of an __init__-less subdirectory named ‘somepkg’, then pkg_resources already supports mixed-mode installations under PEP 420, and doesn’t even need to be updated!

I haven’t checked whether that’s the case, but if it is, then the only thing that setuptools neds to change is its .pth generation magic, to not do the magic if it’s on a PEP 420 platform at runtime, and to stop including __init__.py’s for namespace packages.

So checking if it’s the case:

$ mkdir foo
$ touch foo/text.txt
$ python
Python 3.6.0b1+ (3.6:b0c56ee58478, Sep 24 2016, 09:12:37) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pvd = pkg_resources.get_provider('foo')
>>> pvd
<pkg_resources.NullProvider object at 0x10c9ec470>
>>> pvd.loader
<_frozen_importlib_external._NamespaceLoader object at 0x109fab978>

So it seems that at least nominally pkg_resources is able to provide resources for the trivial package. Although,

>>> pvd.get_resource_string('foo', 'text.txt')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1457, in get_resource_string
    return self._get(self._fn(self.module_path, resource_name))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1532, in _get
    "Can't perform this operation for loaders without 'get_data()'"
NotImplementedError: Can't perform this operation for loaders without 'get_data()'

I suspect that if the package had some metadata (dist-info or egg_info), it would also work. So I want to say PJE’s supposition is tenatively confirmed and focus on

the only thing that setuptools [needs] to change is its .pth generation magic, to not do the magic if it’s on a PEP 420 platform at runtime

Other interesting details in this thread.

Sadly, this not only affects --target, but also --prefix… for now settled with the suggestion of @Ivoz, but this is ugly as hell:

for i in $(find lib -name site-packages -print); do echo 'import site, os.path; site.addsitedir(os.path.dirname(__file__))' > $i/sitecustomize.py; done

pip really should do something about it.

You could manually move the namespace’d package that you want to the actual folder where it would be located if it was simply part of the overall package, in a build / deploy step.

Alternately, call addsitedir explicitly in your own sitecustomize.py file.