pyproject-hooks: Version 1.2 breaks pypa/build on Python 3.8 and 3.9

1.2 has broken build’s test suite when using backend-path on Python 3.8 and 3.9 (and check-sdist’s setuptools test, though I just dropped the setuptools test there). See https://github.com/pypa/build/pull/820, https://github.com/python/importlib_metadata/issues/508, https://github.com/henryiii/check-sdist/pull/55. I’ve finally narrowed it down to this package’s 1.2 release.

Backend path is ".", and setuptools is making test_no_prepare.egg-info, which is then getting picked up as a nameless distribution, which causes a confusing error when trying to get the name and falling back on None, then not handling the None case.

_Originally posted by @henryiii in https://github.com/pypa/pyproject-hooks/issues/199#issuecomment-2395287934_

Traceback (most recent call last):
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/compat/py39.py", line 19, in normalized_name
    return dist._normalized_name
AttributeError: 'PathDistribution' object has no attribute '_normalized_name'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/build/.tox/py39/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/build/.tox/py39/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/build/.tox/py39/lib/python3.9/site-packages/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
    return _build_backend().build_wheel(
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/build_meta.py", line 421, in build_wheel
    return self._build_with_temp_dir(
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/build_meta.py", line 403, in _build_with_temp_dir
    self.run_setup()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 1, in <module>
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 183, in setup
    return run_commands(dist)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
    dist.run_commands()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/bdist_wheel.py", line 433, in run
    self.run_command("install")
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/install.py", line 91, in run
    return super().run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/command/install.py", line 706, in run
    self.run_command(cmd_name)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/install_egg_info.py", line 32, in run
    self.run_command('egg_info')
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/build/.tox/py39/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 302, in run
    for ep in metadata.entry_points(group='egg_info.writers'):
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 1051, in entry_points
    return EntryPoints(eps).select(**params)
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 1048, in <genexpr>
    eps = itertools.chain.from_iterable(
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/_itertools.py", line 17, in unique_everseen
    k = key(element)
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/compat/py39.py", line 23, in normalized_name
    return Prepared.normalize(getattr(dist, "name", None) or dist.metadata['Name'])
  File "/build/.tox/py39/lib/python3.9/site-packages/importlib_metadata/__init__.py", line 889, in normalize
    return re.sub(r"[-_.]+", "-", name).lower().replace('-', '_')
  File "/usr/local/lib/python3.9/re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 20 (16 by maintainers)

Most upvoted comments

@jaraco, I suspect that this parish aspect is an issue in setuptools because it uses entry-points to generate the .egg-info directory itself. So setuptools create an empty .egg-info folder, and then it fails while populating it (race-condition-ish unravelled by the change).

In that case, maybe the issue could be addressed in Setuptools by keeping the egg-info directory off sys.path until it’s populated.

That said, an egg-info directory containing no metadata is in fact invalid, right? Shouldn’t the solution be to address the root cause (prevent that invalid metadata from existing)? What is the origin of these empty metadata directories?

@jaraco, I suspect that this parish aspect is an issue in setuptools because it uses entry-points to generate the .egg-info directory itself. So setuptools create an empty .egg-info folder, and then it fails while populating it (race-condition-ish unravelled by the change).

Oh, do we understand why this issue only affects Python 3.9 and earlier? If the underlying problem is a distribution with no Name, I’d expect that also to cause problems. I guess it doesn’t because Distribution.name (or ._normalized_name) happens to return None for these invalid distributions and because the unique check when scanning entry points happens not to care if it gets None where it expects a str. That means that this issue will also affect Python 3.14, where importlib.metadata.Distribution.name will now raise a KeyError when no name is present in the metadata.

It would appear #199 returns a stdlib (importlib.metadata) distribution which setuptools then attempts to operate on using the backport (importlib_metadata) and the two are incompatible on Python 3.8 and .9.

As a reminder, importlib.metadata prior to Python 3.10 was considered “provisional”, so it’s probably preferable to use importlib_metadata at least on those versions.

importlib_metadata has attempted to accommodate objects from those implementations, but it’s complicated due to the two API surfaces that library manages (providers and consumers). In python/importlib_metadata#486, we’re exploring ways to shore up some of those relationships, but that still won’t fix the fact that importlib.metadata on Python 3.9 and earlier will never honor the newer interfaces and guarantees of later releases. Later versions of importlib_metadata will provide an error earlier when the Name key is missing, but only for Distributions constructed from importlib_metadata.

That said, an egg-info directory containing no metadata is in fact invalid, right? Shouldn’t the solution be to address the root cause (prevent that invalid metadata from existing)? What is the origin of these empty metadata directories?

This is only about Python 3.8 and 3.9, everything works fine after that.

It breaks both:

$ git clone https://github.com/cvxpy/cvxpy
$ cd cvxpy
$ pipx run --python 3.9 build --sdist
...
TypeError: expected string or bytes-like object
$ pipx run --python 3.9 build --wheel
...
TypeError: expected string or bytes-like object
$ pipx run --python 3.10 build --sdist
...
Successfully built cvxpy-1.6.0.tar.gz

If you avoid the new pyproject-hooks, it works:

$ uvx --python 3.9 --from build --with pyproject-hooks==1.1.0 pyproject-build --sdist
...
Successfully built cvxpy-1.6.0.tar.gz

You have to remove the *.dist-info dir after a successful build, however, otherwise it starts working.

My inclination is to say that this is either a bug in setuptools (which should spot when it gets a stdlib object and deal with it) or in the importlib_metadata backport (which should offer some sort of compatibility).

None of which helps that much, unless those projects can release fixes ASAP, but at the same time, I sympathise with @takluyver’s point that if we can’t rely on being able to safely pass distribution objects around, we’re going to be at risk of yet more areas of incompatibility popping up.