poetry: setup.py generated by `poetry build` does not have included file in package_data

i hit the same when i try to include a marker according to PEP561:

[tool.poetry]
# …
include = ["mypackage/py.typed"]
# …

which results in

# …
package_data = \
{'': ['*']}
# …

but the file is not excluded in any vcs-related context.

I’m having the same issue.

@abn just wanted say a big THANK YOU! I really appreciate the well thought out responses. Poetry is in good hands.

i hit the same when i try to include a marker according to PEP561:

[tool.poetry]
# …
include = ["mypackage/py.typed"]
# …

which results in

# …
package_data = \
{'': ['*']}
# …

but the file is not excluded in any vcs-related context.

py.typed now appears to be properly included after packaging my project with Poetry 1.1.0 and poetry-core 1.0.0.

# pyproject.toml
[tool.poetry]
include = ["mypackage/py.typed"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

To verify without affecting your PyPI package, you can publish to Test PyPI as described in the PyPA tutorial.

Thank you to @abn and the other maintainers and contributors 🎉 👏

@atollk I would not call the use of setup.py to be the “canonical way” of managing python packages. It has most certainly been the defacto option for a long while. However with the introduction of PEP 517 and PEP 518, the python packaging ecosystem is building towards a “standards” based approach that detaches the tool specifics from the package build-system configuration. The use of setup.py simply indicates the use of setuptools as the PEP 517 build backend. For poetry managed packages we expect poetry to be used as the build backend. For context, pip supports this from version 19.1. TLDR; python package managers should be able to build packages defining build systems via pyproject.toml.

As for PEP 561, by the time the setup file generation is removed, we anticipate the required features to be supported. For example see #2000. This needs to be ported to core, but I do expect to see the py.typed usecase supported soon. Hope that helps.

I’m not sure how related my problem is to this issue and if it warrants a separate issue, but include = ['package/file.py'] doesn’t seem to override my .gitignore as I expected it to. It does not end up in my installed package in site-packages when I run pip install ..

.gitignore:

package/file.py

pyproject.toml:

[tool.poetry]
include = ["package/file.py"]
...

$ pip install .
(installs successfully)
$ ls .../site-packages/package/
...
(no file.py)

Removing the entry from .gitignore works fine, however.

Running poetry build seems to correctly include the file in the wheel and sdist (and doesn’t if I don’t include the tool.poetry.include entry, as expected).

I’m working on it in https://github.com/ActivityWatch/aw-qt/pull/51

Edit: I made a dirty workaround that temporarily comments out the file from .gitignore before building (and uncomments when build is done), but would appreciate a clean fix.

removing setup.py from sdist builds is a backwards incompatible change.

@bdoms Yes, we understand that in rare cases where the project maintainer wants to remain compatible with downstream build scenarios that rely on older build toolchain, this might require user intervention. However, supporting a feature like this in perpetuity is not a good idea for a few reason. Maintaining setup.py generation as we expand poetry’s capabilities becomes harder as it requires us to take into consideration various cases. This overhead does not make sense in comparison with the small cross section of scenarios that may be invalidated (ie. package manager, pip or otherwise, uses a version that does not understand PEP-517 and only knows setuptools, the platform does not have pre-built wheels for the packages and requires the use of the sdist) and the fact that these issues can easily be worked around. Also it should be noted that the generation of setup.py file was always meant to be a workaround rather than a supported feature. The current implementation of this is in no way considered reliable in all scenarios. Additionally, with poetry>=1.1.0, if the build happens in a poetry managed virtual environment, the pip version used will be the version bundled with virtualenv project, which at the moment is 20.x.

All this said, it is also important to note that this is not happening overnight. The following steps need to happen prior to this being removed.

1. tool.poetry.build.generate-setup-file introduced in a GA release (1.1.0), defaulting to true. (opt-in)
2. tool.poetry.build.generate-setup-file defaults to false (1.2.0). (opt-out)
3. setup.py generation deprecation (unplanned).
4. setup.py generation removal (unplanned).

I like Poetry a lot because it has generally made my life easier, but removing setup.py from sdist builds is a backwards incompatible change. There’s no guarantee that people are using a version of pip that works without it, or even pip or any other build tool at all - they may just expect setup.py to be there and call it directly. If you’re following semantic versioning the backwards incompatibility requires a major version change in order to be implemented. I get that you may not use it yourself, but make no mistake: THIS WILL BREAK PEOPLE’S WORKFLOWS. There’s gonna be some bad days when build/deploy systems suddenly stop working and devs are scrambling to find out why.

As for whether it’s still happening, I just ran poetry self update on Ubuntu 20.04 with Python 3.8 and the Poetry version I got was 1.0.9. In that version, yes, this is still happening just as described in the initial report above.

@sdispater thanks for checking in! I upgraded to 1.0.0b5 and this is still a problem: a file I had excluded in .gitignore but explicitly included in pyproject.toml is there in the tar but not in setup.py.

The code has moved around a bit since I last checked, but looking at it now the problem appears to be this line: https://github.com/sdispater/poetry/blob/master/poetry/masonry/builders/sdist.py#L137

It’s the else case for the if isinstance(include, PackageInclude) check when creating the list of files that becomes the package_data. I have no idea why the files I need fail that check, but all of the files I’ve been having problems with fail there, so they hit the pass line, and thus are not included in the list.