python-buildpack: Fails to handle PEP-518 build dependencies - cannot install vendored alembic==1.4.2

What version of Cloud Foundry and CF CLI are you using? (i.e. What is the output of running cf curl /v2/info && cf version?

{
   "name": "",
   "build": "",
   "support": "",
   "version": 0,
   "description": "Cloud Foundry at SAP Cloud Platform",
   "api_version": "2.146.0",
   "osbapi_version": "2.15",
}

cf version 6.46.0+29d6257f1.2019-07-09

What version of the buildpack you are using?

 1.7.10

If you were attempting to accomplish a task, what was it you were attempting to do?

I am trying to push an app with vendored dependencies on Python 3.7; including alembic==1.4.2. This worked without a problem in Python 3.6 using pip 18.x and breaks with Python 3.7 using pip 19.x.

Please find reproduction steps in this POC repository: https://github.com/mhaas/cf-pep517-repro Simply execute run.sh. Note that the script will ask to delete the app; so watch the screen.

What did you expect to happen?

I expected the cf push to succeed both on Python 3.6 and on Python 3.7.

What was the actual behavior?

The push fails with Python 3.7.

The new version of pip is now always trying to build a wheel before installing. The old version would check if wheel was installed and, if it was not installed, simply fall back to setup.py install. The current behavior of building a wheel fails due to the use of --no-build-isolation in the buildpack, which prevents automatic installation of the wheel package. The wheel package is not preinstalled, hence installation of the package fails completely.

Note that in my POC, the build eventually succeeds because the buildpacks falls back to the non-vendored code path. This is not the case for my actual scenario, since I have some private dependencies. In my actual scenario, cf push fails here.

My Analysis:

I have created a related issue in the pip tracker because I consider this also a regression in pip.

Nevertheless, I also consider this a bug in this buildpack. Some thoughts:

• Documentation for pip 20.1.1 states: Disable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if this option is used
  • PEP-518 specifies ["setuptools", "wheel"] as a reasonable default.
  • The wheel package is missing in the buildpack
• I am not sure about the overall usefulness of --no-build-isolation.
  • My assumption is that this used to prevent the download of non-vendored packages. However, the buildpack will fall back to the non-vendored path in case of error and will in any case download packages from the net. So, the flag does not seem to be useful to prevent network access.
• Use of --no-use-pep517 for will break packages actually using PEP-517, so this is also not a viable workaround
• IF someone actually vendored the build dependencies (setuptools, wheel) into vendor/, the use of --no-build-isolation would actually break that.
  • Note that pip download currently does not download build-time dependencies: https://github.com/pypa/pip/issues/7863

When omitting --no-build-isolation, the install will attempt to install build depencencies from vendor/:

$ pip install -r requirements.txt  --ignore-installed --exists-action=w --no-index --find-links=$(pwd)/vendor
Looking in links: /mhaas/cf-pep517-repro/vendor
Collecting alembic==1.4.2 (from -r requirements.txt (line 1))
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /mhaas/cf-pep517-repro/venv-3.7/bin/python3.7 /mhaas/cf-pep517-repro/venv-3.7/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/_j/hnv2m0816dj2rr3bq7nxz92r0000gn/T/pip-build-env-6pqv1yr6/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /mhaas/cf-pep517-repro/vendor -- 'setuptools>=40.8.0' wheel
       cwd: None
  Complete output (4 lines):
  Looking in links: /Users/d073668/work/github.com/mhaas/cf-pep517-repro/vendor
  Collecting setuptools>=40.8.0
    ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)
  ERROR: No matching distribution found for setuptools>=40.8.0

With --no-build-isolation, the installation will not be attempted:

(venv-3.7) ➜  cf-pep517-repro git:(master) ✗ pip install -r requirements.txt --no-build-isolation  --ignore-installed --exists-action=w --no-index --find-links=$(pwd)/vendor
Looking in links: /mhaas/cf-pep517-repro/vendor
Collecting alembic==1.4.2 (from -r requirements.txt (line 1))
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
[...]
    error: invalid command 'bdist_wheel'

This uses the installation command line taken from the buildpack itself.

I admit this is a bit of mess, also on the side of pip and python package management in general. The following solutions come to mind:

1. Just install wheel 1.1 Hope that the installed version is good enough and that no package comes along with either a setup_requires or a PEP-518 build-time dependency which cannot be satisfied by the pre-installed versions of setuptools and/or wheel
2. Just get rid of --no-build-isolation 2.1 This option completely breaks build-time dependency management 2.2 Instead, Rely on user to vendor also build-time dependencies. This is currently not well supported by pip, but is less reliant on prayers than solution 1.
3. Hybrid approach: 3.1 Get rid of --no-build-isolation flag AND ship copies of wheel and setuptools wheels inside the buildpack. In addition to pointing to vendor, also add the local repository containing wheel and setuptools to make a reasonable attempt to satisfy these dependencies 3.2 The local repository is required here, as pip will attempt to create isolate build environments for each package and install the build dependencies from scratch. This means that pre-installed versions of setuptools and wheel are ignored here

Finally, I am not sure how the whole issue will behave in case of legacy setup.py packages which use build-requires.

Just for reference; PEP-517/PEP-518 define a newer style of packaging which allows declaration of build-time dependencies as well as pluggable packaging frontends. As this becomes more frequent (even using non-standard dependencies), I expect that more issues like this one will come back when running the buildpack in vendored mode.

Sorry for the wall of text - I spent some time debugging this. Thanks for reading this far!

Please confirm where necessary:

• I have included a log output
• My log includes an error message
• I have included steps for reproduction