flit: Bug: cyclic dependency with `tomli==1.2.1` and `flit_core==3.2.0`

Looks like if you use a version of pip < 21.x , installs get hosed.

pip 19 yells at you about not supporting pep 517:

  pip._vendor.pep517.wrappers.BackendUnavailable
  You are using pip version 19.0.3, however version 21.3 is available.
  You should consider upgrading via the 'pip install --upgrade pip' command.

pip 20 - not so much (as seen in the above comment ) - it tries and fails.

pip>=21 has no issue.

Upgrading the pip version resolves the dataflow flex-template issue.

Hello - we are using GCP Dataflow using python and we are seeing an error that is precisely related to this cyclic dependency. This breaks us unfortunately - and it is not easy for us to change the way it wants to run (because this is GCP’s code and not our code).

We could do, but poetry-core bundles quite a few packages, so it doesn’t seem like the best starting point for bootstrapping. And if you unbundle those, some of them are built with setuptools, so we’d be chaining back to setuptools again, which I’d like to demonstrate isn’t necessary. And setuptools has its own cycle if you use the PEP 517 APIs (setuptools <-> wheel).

Another option is that we could make a special backend to build tomli (either in flit_core.build_tomli or in tomli’s own source tree) something like this:

import sys
sys.path.insert(0, '.')
from flit_core.buildapi import *

That would have the same effect as the manual workaround I’ve suggested for downstream packagers. But it wouldn’t solve the issues people are having using pip, because flit_core would still have a dependency on tomli.

I believe that the packaging ecosystems that are strict about building everything from source also tend to strongly object to bundling, so I don’t think a solution that relies on bundling a TOML parser in the build backend will make them happy. Even if they have streamlined unbundling in general, unbundling in this case just puts you back in the same dependency cycle (whether that’s flit_core <-> tomli, or poetry_core <-> tomlkit). So it’s going to need some kind of special intervention.

Maybe I’m wrong about this, though. 🙂 If there are significant use cases where people need to install from source but are fine with bundled dependencies, we can do that (either rely on Poetry, or bundle tomli in flit_core sdists).

My impression is that a lot of the people having issues don’t actually need to build everything from source, but are doing that as a convenient shorthand for building something from source. It would be nice if pip had better ways to control this (e.g. ‘build from source anything that involves native code’ - but I don’t think that’s knowable ahead of time.)

We are using Dataflow’s Flex Templates (https://cloud.google.com/dataflow/docs/guides/templates/using-flex-templates) As a part of this, we build our own container image based off a GCP base image (of which we do not have any direct access to).

When the Dataflow job kicks off, the Apache beam program runs and it starts off by installing some python packages. This is where we are seeing the error - and it exits. This started failing yesterday, which coincided with the new flit_core release. Also, the log message below indicates the problem. A lot of these packages are required - and not specified by code in my control.

Anything you can do would be much appreciated - thanks!

Output from execution of subprocess: b'Collecting google-cloud-secret-manager==2.3.0
  Downloading google-cloud-secret-manager-2.3.0.tar.gz (90 kB)
  Saved /tmp/dataflow-requirements-cache/google-cloud-secret-manager-2.3.0.tar.gz
Collecting google-api-core[grpc]<2.0.0dev,>=1.22.2
  Downloading google-api-core-1.31.3.tar.gz (97 kB)
  Saved /tmp/dataflow-requirements-cache/google-api-core-1.31.3.tar.gz
Collecting grpc-google-iam-v1<0.13dev,>=0.12.3
  Downloading grpc-google-iam-v1-0.12.3.tar.gz (13 kB)
  Saved /tmp/dataflow-requirements-cache/grpc-google-iam-v1-0.12.3.tar.gz
Collecting proto-plus>=1.4.0
  Downloading proto-plus-1.19.4.tar.gz (48 kB)
  Saved /tmp/dataflow-requirements-cache/proto-plus-1.19.4.tar.gz
Collecting libcst>=0.2.5
  Downloading libcst-0.3.21.tar.gz (576 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qhpytueu/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools_scm
       cwd: None
  Complete output (69 lines):
  Collecting setuptools_scm
    Downloading setuptools_scm-6.3.2.tar.gz (57 kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-9hty5unr/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=45' wheel 'tomli>=1.0' 'packaging>=20.0'
         cwd: None
    Complete output (57 lines):
    Collecting setuptools>=45
      Using cached setuptools-58.2.0.tar.gz (2.3 MB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Collecting wheel
      Using cached wheel-0.37.0.tar.gz (65 kB)
    Collecting tomli>=1.0
      Downloading tomli-1.2.1.tar.gz (14 kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'
      ERROR: Command errored out with exit status 2:
       command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-zp5sz97w/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
           cwd: None
      Complete output (35 lines):
      Collecting flit_core<4,>=3.2.0
        Downloading flit_core-3.4.0.tar.gz (27 kB)
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
          Preparing wheel metadata: started
          Preparing wheel metadata: finished with status 'done'
      Collecting tomli
        Using cached tomli-1.2.1.tar.gz (14 kB)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
          status = self.run(options, args)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
          return func(self, options, args)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 324, in run
          reqs, check_supported_wheels=not options.target_dir
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
          discovered_reqs.extend(self._resolve_one(requirement_set, req))
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
          abstract_dist = self._get_abstract_dist_for(req_to_install)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/resolution/legacy/resolver.py", line 340, in _get_abstract_dist_for
          abstract_dist = self.preparer.prepare_linked_requirement(req)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 483, in prepare_linked_requirement
          req, self.req_tracker, self.finder, self.build_isolation,
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 90, in _get_prepared_distribution
          with req_tracker.track(req):
        File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
          return next(self.gen)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
          self.add(req)
        File "/usr/local/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 114, in add
          raise LookupError(message)
      LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli>=1.0 from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442
      WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
      You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
      ----------------------------------------
    ERROR: Command errored out with exit status 2: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-zp5sz97w/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4' Check the logs for full command output.
    WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
    You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-9hty5unr/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=45' wheel 'tomli>=1.0' 'packaging>=20.0' Check the logs for full command output.
  WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
  You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qhpytueu/normal --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- setuptools_scm Check the logs for full command output.
WARNING: You are using pip version 20.2.4; however, version 21.2.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

Should build move to flit (see https://github.com/pypa/build/issues/394), then to bootstrap Python you would only need five packages: flit_core, tomli, pep517, build and installer. If tomli switches to poetry-core, that’s going to balloon to at least fourteen, not counting their build dependencies, assuming unbundling.

I agree, my use case is not that of a normal user. But even bootstrapping does not help: https://github.com/hukkin/tomli/issues/130#issuecomment-939863846

This actually broke the release-pipeline for a product in maintenance. Manual intervention was difficult but possible. Lockfiles don’t help here, as pip download does not respect them during the “build-dependencies” setup.

My only wish is for a little stricter dependency constraints so this does not happen again.

Feel free to close.