setuptools: Updating MANIFEST.in does not correctly update the package sdist creates

Originally reported by: spookylukey (Bitbucket: spookylukey, GitHub: spookylukey)

The behaviour of sdist depends on previous contents of MANIFEST.in, not just the current. This is not fixed even by running setup.py clean or setup.py clean --all (although this should not be necessary).

This is very surprising behaviour, and potentially dangerous too - if someone accidentally adds a MANIFEST.in rule that includes a file that must not be distributed and notice the problem, they would expect that removing the rule will remove the file, but it does not.

I’ve attached a bash script that demonstrates the problem.

Bitbucket: https://bitbucket.org/pypa/setuptools/issue/436

About this issue

Original URL
State: open
Created 9 years ago
Reactions: 9
Comments: 20 (13 by maintainers)

Commits related to this issue

Merge pull request #1014 from miccoli/issue-436 addresses #436: Updating MANIFEST.in does not correctly update the package sdist creates — committed to pypa/setuptools by jaraco 7 years ago
Revert "addresses #436: Updating MANIFEST.in does not correctly update the package sdist creates" — committed to miccoli/setuptools by miccoli 7 years ago
Revert "addresses #436". Fixes #1016. This reverts commit 1955e5b0df67cc1aa389b8c655199958a6fcc6a0. — committed to pypa/setuptools by jaraco 7 years ago
setup.py: fix sdist if files are removed setuptools has a long-standing bug that if files are removed from the list of sources but were included in a previous run of egg_info, they remain in the gene... — committed to osandov/drgn by osandov 5 years ago
setup.py: fix sdist if files are removed setuptools has a long-standing bug that if files are removed from the list of sources but were included in a previous run of egg_info, they remain in the gene... — committed to osandov/drgn by osandov 5 years ago
setup.py: fix sdist if files are removed setuptools has a long-standing bug that if files are removed from the list of sources but were included in a previous run of egg_info, they remain in the gene... — committed to osandov/drgn by osandov 5 years ago

Most upvoted comments

I really think SOURCES.txt should be cleared automatically at some point. This is very surprising behavior. I’ve been trying to figure out why the changes I was making to MANIFEST.in didn’t seem to be working.

rsyring on Dec 21, 2018

I have been hit by this same bug. My current remedy is to delete the *.egg-info directory before running setup.py sdist.

miccoli on Feb 27, 2017

@lucacerone This issue is specific to a setup with MANIFEST.in: please first check the docs at https://packaging.python.org/guides/using-manifest-in/ and see if following those instruction you are able to obtain the desired source distribution.

The only thing that you should be aware, regarding this specific bug, is that after each change to MANIFEST.in you should delete the <package>.egg-info directory to be sure that the SOURCES.txt is regenerated appropriately.

miccoli on Apr 24, 2020

The reason for

        rcfiles = list(walk_revctrl())
        if rcfiles:
            self.filelist.extend(rcfiles)
        elif os.path.exists(self.manifest):
            self.read_manifest()

is now clear: in the develop tree (under SCM control) the files to be installed are determined from the SCM system (walk_revctrl()); in an sdist install the files are to be determined from the existing manifest file, which was previously generated in the develop tree + setuptools_scm

This logic is broken for packages that do not use setuptools_scm, for which there is no guarantee that the current manifest file (in the development tree) is correct.

Unfortunately my PR #1014 badly brokes this intended behaviour, so that pip install of sdist packages under SCM control and include_package_data=True is not more possible, see #1016 .

miccoli on Apr 18, 2017