pip-tools: Mysterious error with optional dependencies in pyproject.toml

Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with

[tool.setuptools]
py-modules = []

---

Aside from that I’ll note that the alternative runner selection is now in build, but there has not yet been a release including it.

I’ve stumbled upon this issue a couple of times in the last month. Here is the error I see:

$ pip-compile pyproject.toml -v
Creating venv isolated environment...
Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
Getting build dependencies for wheel...
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/OSS/opensearch-logger/pyproject.toml

Setup

Environment:

• Python 3.11
• MacOS 13.0.1
• arm64

The pyproject.toml file itself is super simple and passes validation using the validate-pyproject package:

[build-system]
requires = [
    "flit_core >=3.2,<4",
]
build-backend = "flit_core.buildapi"

[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]

What causes it

It’s definitely not a pip-compile problem as it just calls project_wheel_metadata from the build package and then catches the error and reports.

https://github.com/jazzband/pip-tools/blob/eff84765725fc15579d6626851910350d580ec8b/piptools/scripts/compile.py#L474-L476

The build package, in turn, does lots of stuff and ends up calling a Python script called _in_process.py from the pep517 package.

I have no idea how it is supposed to work, but it looks like the build package create a one-time temporary virtual environment in which this _in_process.py script is then invoked. Here is the command with which build package tries to call a subprocess in my case:

['/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-rj6bf2id/bin/python', '/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpb6wdunjz']

Obviously, when something wrong happens in that subprocess script all we get at the end is:

Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /Users/vduseev/Projects/Playground/pip-tools-bug/pyproject.toml

But in reality, the real error does not ever get propagated to the parent process and we don’t see it. I’ve caught it by instrumenting the _in_process.py file in my virtual environment (lib/python3.11/site-packages/pep517/in_process/_in_process.py) with print statements.

Exception: description must be specified under [project] or listed as a dynamic field
_in_process.py was called with args: ['/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/tmpttsp4wmf']
Exception: description must be specified under [project] or listed as a dynamic field
  File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 338, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vduseev/Projects/Playground/pip-tools-bug/.venv/lib/python3.11/site-packages/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/buildapi.py", line 23, in get_requires_for_build_wheel
    info = read_flit_config(pyproj_toml)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 79, in read_flit_config
    return prep_toml_config(d, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 106, in prep_toml_config
    loaded_cfg = read_pep621_metadata(d['project'], path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/59/45422z7x04vc6pztgy7xm51m0000gn/T/build-env-tpnuh3w_/lib/python3.11/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
    raise ConfigError(

Now I can see that the underlying error actually comes from the flit package, complaining about lack of the description field in the [project] section of my pyproject.toml file.

How to get the same output

Here is what I manually added to the _in_process.py file in pep517 package in my venv

https://github.com/pypa/pyproject-hooks/blob/4c9325f4e594bfc7178452d0d01eb8da6c3dbbdb/src/pyproject_hooks/_in_process/_in_process.py#L322

def main():
    # begin: added
    from pathlib import Path
    log_file = Path(__file__).parent / "log.txt"
    with log_file.open("a") as f:
        try:
    # end: added
            if len(sys.argv) < 3:
                sys.exit("Needs args: hook_name, control_dir")
            hook_name = sys.argv[1]
            control_dir = sys.argv[2]
            if hook_name not in HOOK_NAMES:
                sys.exit("Unknown hook: %s" % hook_name)
            hook = globals()[hook_name]

            hook_input = read_json(pjoin(control_dir, 'input.json'))

            json_out = {'unsupported': False, 'return_val': None}
            try:
                json_out['return_val'] = hook(**hook_input['kwargs'])
            except BackendUnavailable as e:
                json_out['no_backend'] = True
                json_out['traceback'] = e.traceback
            except BackendInvalid as e:
                json_out['backend_invalid'] = True
                json_out['backend_error'] = e.message
            except GotUnsupportedOperation as e:
                json_out['unsupported'] = True
                json_out['traceback'] = e.traceback
            except HookMissing as e:
                json_out['hook_missing'] = True
                json_out['missing_hook_name'] = e.hook_name or hook_name

            write_json(json_out, pjoin(control_dir, 'output.json'), indent=2)
        # begin: more_added
        except Exception as e:
            f.write(f"Exception: {e}\n")
            import traceback
            traceback.print_tb(e.__traceback__, file=f)
        # end: more_added

Fixing the issue

I can address the complaints of flit by adding missing things:

• Adding description field
• Making sure a module opensearch_logger exists in the same directory where pyproject.toml is placed

And voila:

#
# This file is autogenerated by pip-compile with python 3.11
# To update, run:
#
#    pip-compile pyproject.toml
#
certifi==2022.9.24
    # via
    #   opensearch-py
    #   requests
charset-normalizer==2.1.1
    # via requests
idna==3.4
    # via requests
opensearch-py==2.0.0
    # via opensearch-logger (pyproject.toml)
requests==2.28.1
    # via opensearch-py
urllib3==1.26.13
    # via
    #   opensearch-py
    #   requests

Alternatively, should I swap the build system to classic setuptools, it compiles successfully even without said fixes.

[build-system]
requires = [
    "setuptools",
    "wheel",
]
build-backend = "setuptools.build_meta"

[project]
name = "opensearch-logger"
version = "1.2.1"
requires-python = ">=3.6"
dependencies = ["opensearch-py"]

Conclusion

I don’t understand why such system was chosen by creators of build system (not pip-tools). This is the reason we don’t see an underlying problem from the chosen build system. It just does not get propagated to higher level tools, such as build or pip-tools.

P.S. was so happy to see @atugushev in the comments of this issue. Cheers!

Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with

[tool.setuptools]
py-modules = []

Aside from that I’ll note that the alternative runner selection is now in build, but there has not yet been a release including it.

I also experienced this problem. I have a valid pyproject.toml with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre’s solution above is what worked for me. Thanks @AndydeCleyre!

FTR: tracking issue https://github.com/pypa/build/issues/553 for project_wheel_metadata improvement.

@vduseev thanks for chiming in and for the additional context!

Digging into the issue I found that build.utils.project_wheel_metadata uses pep517.quiet_subprocess_runner which suppresses output. With pep517.default_subprocess_runner It shows the actual error.

Patched build:

diff --git src/build/util.py src/build/util.py
index 9675393..a435479 100644
--- src/build/util.py
+++ src/build/util.py
@@ -43,7 +43,7 @@ def project_wheel_metadata(
     """
     builder = build.ProjectBuilder(
         os.fspath(srcdir),
-        runner=pep517.quiet_subprocess_runner,
+        runner=pep517.default_subprocess_runner,
     )

     if not isolated:

pip-compile output:

root@35878ab01f42:/foo# pip-compile pyproject.toml
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 351, in <module>
    main()
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 333, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/usr/local/lib/python3.10/site-packages/flit_core/buildapi.py", line 47, in prepare_metadata_for_build_wheel
    ini_info = read_flit_config(pyproj_toml)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 79, in read_flit_config
    return prep_toml_config(d, path)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 106, in prep_toml_config
    loaded_cfg = read_pep621_metadata(d['project'], path)
  File "/usr/local/lib/python3.10/site-packages/flit_core/config.py", line 630, in read_pep621_metadata
    raise ConfigError(
flit_core.config.ConfigError: description must be specified under [project] or listed as a dynamic field
Backend subprocess exited when trying to invoke prepare_metadata_for_build_wheel
Failed to parse /foo/pyproject.toml

---

@pradyunsg I wonder if would it be okay to introduce one of the following params in build.utils. project_wheel_metadata()? For example:

• project_wheel_metadata(quiet=False) - do not suppress the output
• project_wheel_metadata(runner=my_runner) - use my runner

A colleague has tracked down the source of the problem. If you check pyproject.toml with validate-pyproject you get:

[ERROR] project.authors[{data__authors_x}] must be object

Putting the authors field into the correct format fixes things, i.e.:

authors = [{ name = "author1", email = "author1@email.com" }]

I’ve dug in a bit more and it’s weirder than I thought. This following pyproject.toml doesn’t work:

[project]
name = "SolidityGUI"
version = "0.1"
authors = ["author1"]
dependencies = ["pygmsh", "pyside6", "vtk"]

[project.optional-dependencies]
dev = ["black"]

However, if you remove the authors line, it works fine again:

[project]
name = "SolidityGUI"
version = "0.1"
dependencies = ["pygmsh", "pyside6", "vtk"]

[project.optional-dependencies]
dev = ["black"]

How odd is that?!

I found another case today where this error can occur:. I started with a working pyproject.toml, then I created two subdirectories.

The clue for me was that pip install --editable . gave an error “multiple top-level packages discovered in a flat-layout”.

Running validate-projected returned no errors.

$ cat pyproject.toml 
[project]
name = 'test'
description = 'test'
version = '0.1.0'
authors = [ ]
dependencies = [ 'simplejson ' ]
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
    # via test (pyproject.toml)
$ mkdir foo
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile --resolver=backtracking pyproject.toml
#
simplejson==3.18.4
    # via test (pyproject.toml)
$ mkdir bar
$ ./env/bin/python -mpiptools compile --resolver=backtracking pyproject.toml
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /home/pbrannan/tmp/test-env/pyproject.toml
$ ./env/bin/python -mvalidate_pyproject pyproject.toml
Valid file: pyproject.toml

Ouch, sorry. This is the pip-tools’ message:

https://github.com/jazzband/pip-tools/blob/eff84765725fc15579d6626851910350d580ec8b/piptools/scripts/compile.py#L479

---

pip install .:

      ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
      configuration error: `project.authors[{data__authors_x}]` must be object

---

python -m build:

ValueError: invalid pyproject.toml config: `project.authors[{data__authors_x}]`.
configuration error: `project.authors[{data__authors_x}]` must be object

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

Regarding that last example, it seems to be about the way setuptools auto-discovers the project structure, and can be avoided with

[tool.setuptools]
py-modules = []

Aside from that I’ll note that the alternative runner selection is now in build, but there has not yet been a release including it.

I also experienced this problem. I have a valid pyproject.toml with no syntax errors, running python 3.11 on Ubuntu 22.04. @AndydeCleyre’s solution above is what worked for me. Thanks @AndydeCleyre!

this also solved the problem for me.

So the errors come because pip-compile swallows output of pip install -e . Running the command will reveal the error.

In the case of a flatlayout with multiple packages, mentioned by @cout one can explicitly specify which folders in the flat layout must be included as packages

[tool.setuptools]
packages = ["package1", "package2"]

for the record, build has implemented switching runner from the default quiet one, but it’s not yet included in any releases of build.

pip-compile omitted all error logs, I have to use pip install -e . to see what went wrong with the pyproject.toml file. My problem was wrong syntax for fields license and requires-python

It may still be worth opening a separate issue re having better error reporting in the case where the pyproject.toml file is malformed, but obviously it’s a less important problem.

Agreed. Feel free to open a separate issue on the pip-tools tracker.