setuptools: [BUG] 61.0.0 breaks "setup.py install" by missing dependencies in install_requires

The pip-tools daily scheduled CI began failing four tests 5 days ago, the day of setuptools v61.0.0 release.

Hi @AndydeCleyre, I noticed that the fixtures seem to be creating distributions with no python modules.

When I compare METADATA_TEST_CASES for setup.cfg and setup.py I can see that setup.cfg has packages = find:, but setup.py lacks the equivalent packages=find_packages(). Is that on purpose?

Note that, prior to setuptools v61, these different configurations are not equivalent:

a) The example with setup.cfg would include sample_lib/__init__.py in the distribution b) The example with setup.py would be empty.

For the majority of use cases, the situation in (b) is accidental and unintended (maybe the same accident happened in your test case?).

With the improvements in v61, setuptools will try to auto-detect which files to include, but it will fail if the root directory has a confusing/equivocal layout.

The example generated by the fixtures seems to have extra folders dists and packages.

setuptools>=61 does not know exactly how to deal with these folders and therefore, halts the build process asking for the user to improve the configuration or use a src-layout.

This is the breaking change motivating the major bump from v60 to v61 and is described in the CHANGELOG.

The layouts recognized by setuptools for auto-discovery are described in https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#automatic-discovery.

Layouts that differ from the ones documented are asked to explicitly list packages and/or py_modules in their configuration.

These include “intentionally empty” packages and “meta-packages”[^1]. For those scenarios the users are asked to set packages=[].

[^1]: That only exists to specify dependencies.

Uummm… It is a possibility but I don’t think setuptools is that smart 😅

These are all different ways of “building” the package and how the metadata looks like:

# Assuming the project is cloned somewhere
% cd test-project/parent_folder/pep508-package
% rm -rf src/pep508_package.egg-info build dist && pipx run build
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...
% tar xOf dist/pep508_package-1.0.0.tar.gz pep508_package-1.0.0/src/pep508_package.egg-info/requires.txt
six
sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...


% .venv/bin/python -m pip install -U setuptools
% rm -rf src/pep508_package.egg-info build dist && .venv/bin/python setup.py bdist_wheel
% unzip -c dist/pep508_package-1.0.0-py3-none-any.whl pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...


% .venv/bin/python -m pip uninstall pep508-package
% .venv/bin/python -m pip install 'pep508-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/pep508-package'
% cat .venv/lib/python3.8/site-packages/pep508_package-1.0.0.dist-info/METADATA
...
Requires-Dist: sibling-package @ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
...

It seems that Requires-Dist in the metadata always contain the directory.

@abravalheri I have narrowed my test example down to a single factor, In requirementslib/models/setup_info.py if I modify the method build_pep517 to do this extra hack at the beginning (the hack has knowledge of the package sub-directories`:

def build_pep517(source_dir, build_dir, config_settings=None, dist_type="wheel"):
    if 'pep508_package' in source_dir or 'pep508-package' in source_dir:
        source_dir = f"{source_dir}/parent_folder/pep508-package"

When I have that extra logic disabled, and some extra print statements, I can see that this gets built: parent_folder.sibling_package.src.sibling_package-0.0.0-py3-none-any.whl

build pep517 base_dir /tmp/pipenv-ln7c16i9-src/pep508_package
pep517_config {'--global-option': []}
result of build_pep517: parent_folder.sibling_package.src.sibling_package-0.0.0-py3-none-any.whl

However, with my hack logic in place, the output instead is: pep508_package-1.0.0-py3-none-any.whl

build pep517 base_dir /tmp/pipenv-2xqe9gf8-src/pep508_package
pep517_config {'--global-option': []}
result of build_pep517: pep508_package-1.0.0-py3-none-any.whl

This makes all the difference in the world to get the expected result of:

[{"version": "1.16.0", "hashes": ["sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"], "name": "six", "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2'"}, {"subdirectory": "parent_folder/pep508-package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "name": "pep508-package"}, {"subdirectory": "parent_folder/sibling_package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "name": "sibling-package"}, {"version": "1.16.0", "hashes": ["sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926", "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"], "name": "six", "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'"}, {"subdirectory": "parent_folder/pep508-package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "version": "", "name": "pep508-package"}, {"subdirectory": "parent_folder/sibling_package", "ref": "4f7b2b05e0c99d79ece1e8cfe5411a0e5a478648", "git": "https://github.com/techalchemy/test-project.git", "version": "", "name": "sibling-package"}]

However, I am still head scratching about what could have changed to cause it to no longer automatically get the subdirectory on the path in this step, or why it would have changed behaviors ~4 days ago.

I wonder if it the metadata has empty version and requires because the name is all weird in the new version ‘parent-folder.sibling-package.src.sibling-package’

This is comment intrigued me a little, because it might be related to another change in v61.*.

If you are trying to install parent-folder as if it was a package (note that parent-folder does not contain setup.py or pyproject.toml), previously setuptools would generate a UNKNOWN package, but now with the new name discovery it will indeed create parent-folder.sibling-package.src.sibling-package (the details of this mechanic were first proposed in https://github.com/pypa/setuptools/issues/2887#issuecomment-970708430)

This happens because parent_folder/sibling_package/src/sibling_package is accidentally a valid namespaced package under the eyes of PEP 420. For example, if you are inside the test-project, you can fire up the REPL and successfully run:

>>> import parent_folder.sibling_package.src.sibling_package

I have the impression that the name parent-folder.sibling-package.src.sibling-package is a valid under PEP 508.

I will just add some notes here so when you have the time you can have a look.

I do believe it is caused by setup tools upgrade at this point.

That is completely possible. It might very well be that v61.0.0…v61.1.1 introduced something that is causing the problem. However this seems to be a second issue, and not the one originally reported here.

I think the reason pinning setup tools doesn’t fix it is because internal pip is installing the latest setup tools anyway.

I don’t understand what you mean here. The test I did in my machine does not use pip at all. The original problem mentioned in this issue is related to invoking python setup.py install, so I tried to replicate it locally with the repository you linked. python setup.py install does not use pip (it uses easy_install internally).

If I instead run:

% git clone https://github.com/techalchemy/test-project
% cd test-project/parent_folder/pep508-package
% virtualenv .venv
% .venv/bin/python -m pip install -U pip setuptools==61.1.1 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 61.1.1
wheel      0.37.1
% .venv/bin/python -m pip install -e .
...
  Running command git clone --filter=blob:none --quiet https://github.com/techalchemy/test-project.git /tmp/pip-install-fapb6ksm/sibling-package_2bc442202d024507bd538a882a35d28a
...
Successfully installed pep508-package-1.0.0 sibling_package-1.0.0 six-1.16.0 toml-0.10.2 urllib3-1.26.9
# I also tried `pip install .` and the outcome is the same

Then both packages seem to install fine.

If instead I run (replacing install with develop:

% rm -rf .venv
% .venv/bin/python -m pip install -U pip setuptools==60.10.0 wheel
% .venv/bin/python -m pip list
Package    Version
---------- -------
pip        22.0.4
setuptools 60.10.0
wheel      0.37.1
% .venv/bin/python setup.py develop
...
Couldn't find index page for 'sibling_package' (maybe misspelled?)
...
No local packages or working download links found for sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package
error: Could not find suitable distribution for Requirement.parse('sibling_package@ git+https://github.com/techalchemy/test-project.git@master#subdirectory=parent_folder/sibling_package')

Then I see the same problem with v60.10.0

If you manage to find a minimal reproducer along the lines of the examples before that works for setuptools==60.10.0 but does not work for setuptools==61.1.1 that would help a lot to understand this problem.

BTW, I am testing on:

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0]

Hi @matteius please do enjoy your leisure time, we can have a look at this later 😃