setuptools: install_data doesn't respect "--prefix"

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

This has come up for me in the past, and again recently. I’d like to push back on this to see if we can get this sorted.

Currently we have two mechanisms to install data files in a python package. One of those is package_data, which works consistently and installs things relative to the distribution that is being installed, and the other one of those is data_files which the behavior is… complicated. As of right now if someone has a setup.py that uses distutils they’ll get relative paths in data_files installed relative to sys.prefix. However, if that same person installs that package with easy_install or with pip (w/o wheel) those data_files will stop being installed in that location and will instead be installed relative to the distribution that is being installed. To add even more complexity to this, if someone has a setup.py that uses setuptools they’ll get relative paths in data_files installed relative to the distribution. However if that person installs that package with pip (w/ wheel) then those packages will stop being installed in that location and instead be installed relative to sys.prefix.

I think we need to unify the definition of what data_files actually does because the “well sometimes it means X and sometimes it means Y” is very confusing and, I think, user hostile to the point where I recommend people never to use it if they can help it because it’s mostly just a footgun at this point. My recommendation for people to not use data_files generally works for the kind of data that people can use package_data for, since that does work consistently and it keeps things contained inside of the installed distribution. However, there is other kinds of data that cannot be installed there because it needs to live in specific, well known locations on the system. One example of this is man pages, man pages need to be placed in specific folders or else the man utility cannot find them by default. Currently if a project is installed with distutils or via wheel they can correctly install a man page, but with setuptools there is no such thing available to them.

To that end, I think what we should do is bring setuptools back in line with distutils and with wheel so that all of them install relative paths in data_files relative to sys.prefix. This would technically be a backwards incompatible change but one I think with little impact since the on by default wheel building in pip has already effectively done this for pip’s users and a few projects had to adjust but overall it was pretty painless. I think this makes sense because we cannot change the meaning of data_files in distutils easily, since at the earlier it would land in Python 3.6 and we’d be waiting around for years and years for that to happen. I also think that the distutils defintiion makes more sense in general since we already have the package_data mechanism, having two, slightly different mechanisms for installing data files relative to the installed distribution does not seem like an overall useful thing to me.