pip: pip download with --no-binary flags for projects with alternative pep 517 build systems breaks under some circumstances

Environment

pip version: 18.0
Python version: 3.7.0
OS: Ubuntu 18.0.4

I am in the middle of cutting a pipenv release and I am running the vendoring tooling I ~stole~ borrowed from you guys. This tooling is run via pipenv run invoke vendoring.update which essentially wraps the command in a virtualenv.

Description The failure is happening when downloading licenses, specifically for ptyprocess==0.6.0. The command in question was simply to download the sdist to a target without dependencies. Note that this is functional when disabling build isolation.

This is a ReqTracker related error, which indicates:

LookupError: https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4
c27b0eb6695d846 (from https://pypi.org/simple/requests-download/) is already being built: requests_download from https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24
f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4c27b0eb6695d846 (from flit)

Expected behavior 1 sdist in the target directory sans errors

How to Reproduce

    $ pip download --no-binary :all: --only-binary none --no-deps -d ./__tmp__ ptyprocess==0.6.0

Output

(pipenv)  ³ pipenv-MfOPs1lW  ~/g/pipenv   add-vistir     pip download --no-binary :all: --only-binary none --no-deps -d ./__tmp__ ptyprocess==0.6.0
Collecting ptyprocess==0.6.0                                                                                                               
  Using cached https://files.pythonhosted.org/packages/7d/2d/e4b8733cf79b7309d84c9081a4ab558c89d8c89da5961bf4ddb050ca1ce0/ptyprocess-0.6.0.tar.gz                                                                  
  Saved ./__tmp__/ptyprocess-0.6.0.tar.gz                                                                                           
  Missing build requirements in pyproject.toml for ptyprocess==0.6.0 from https://files.pythonhosted.org/packages/7d/2d/e4b8733cf79b7309d84c9081a4ab558c89d8c89da5961bf4ddb050ca1ce0/ptyprocess-0.6.0.tar.gz#sha256
=923f299cc5ad920c68f2bc0bc98b75b9f838b93b599941a6b63ddbc2476394c0.                                                                            
  This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
  Installing build dependencies ... error                                                                                                                    
  Complete output from command /home/hawk/.virtualenvs/pipenv-MfOPs1lW/bin/python3.7 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-gz0kd7at --no-warn-script-location --no-binary :all: -
-only-binary none -i https://pypi.org/simple -- flit:                                            
  Collecting flit                                                                                                                               
    Using cached https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz
    Missing build requirements in pyproject.toml for flit from https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz#sha256=95b8577b2232da39ee
14ae237575b7a85afeeabc1e87f4a19485fac34f85aa89.                                                                                                                                                                    
    This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
    Installing build dependencies: started                                                                                                                                                                         
    Installing build dependencies: finished with status 'done'                                                                                                                                                     
  Collecting requests (from flit)                                                                                          
    Using cached https://files.pythonhosted.org/packages/54/1f/782a5734931ddf2e1494e4cd615a51ff98e1879cbe9eecbdfeaf09aa75e9/requests-2.19.1.tar.gz                                                                 
  Collecting docutils (from flit)                        
    Using cached https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz                                                                   
  Collecting requests_download (from flit)                                                                                                   
    Using cached https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz                                                         
    Missing build requirements in pyproject.toml for requests_download from https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.
gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4c27b0eb6695d846 (from flit).                                                                                                                            
    This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
    Installing build dependencies: started                                                                                                                                                                         
    Installing build dependencies: finished with status 'error'
    Complete output from command /home/hawk/.virtualenvs/pipenv-MfOPs1lW/bin/python3.7 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qg0tvxfz --no-warn-script-location --no-binary :all:
 --only-binary none -i https://pypi.org/simple -- flit:
    Collecting flit                                                                                                                               
      Using cached https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz
      Missing build requirements in pyproject.toml for flit from https://files.pythonhosted.org/packages/1b/bd/be3cbf0b837dea22ae45bf371879e9caebc6ee064e40f7d7fac7a4c79c45/flit-1.0.tar.gz#sha256=95b8577b2232da39
ee14ae237575b7a85afeeabc1e87f4a19485fac34f85aa89.
      This version of pip does not implement PEP 517 so it cannot build a wheel without 'setuptools' and 'wheel'.
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
    Collecting requests (from flit)
      Using cached https://files.pythonhosted.org/packages/54/1f/782a5734931ddf2e1494e4cd615a51ff98e1879cbe9eecbdfeaf09aa75e9/requests-2.19.1.tar.gz
    Collecting docutils (from flit)
      Using cached https://files.pythonhosted.org/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz
    Collecting requests_download (from flit)
      Using cached https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz
    Exception:
    Traceback (most recent call last):
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/basecommand.py", line 141, in main
        status = self.run(options, args)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 299, in run
        resolver.resolve(requirement_set)                                                                                                                                                                          
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/resolve.py", line 102, in resolve 
        self._resolve_one(requirement_set, req)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one
        abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for
        self.require_hashes
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 297, in prepare_linked_requirement
        with self.req_tracker.track(req):
      File "/home/hawk/.pyenv/versions/3.7.0/lib/python3.7/contextlib.py", line 112, in __enter__
        return next(self.gen)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 74, in track
        self.add(req)
      File "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/lib/python3.7/site-packages/pip/_internal/req/req_tracker.py", line 46, in add
        % (link, fp.read()))
    LookupError: https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4
c27b0eb6695d846 (from https://pypi.org/simple/requests-download/) is already being built: requests_download from https://files.pythonhosted.org/packages/34/9d/431a25538f158a3065a76a6311f40b7908f88a4d24efdbb0ca24
f83bd614/requests_download-0.1.2.tar.gz#sha256=92d895a6ca51ea51aa42bab864bddaee31b5601c7e7e1ade4c27b0eb6695d846 (from flit)

    ----------------------------------------
  Command "/home/hawk/.virtualenvs/pipenv-MfOPs1lW/bin/python3.7 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-qg0tvxfz --no-warn-script-location --no-binary :all: --only-binary none -i
 https://pypi.org/simple -- flit" failed with error code 2 in None

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 2
Comments: 21 (13 by maintainers)

Most upvoted comments

Anyone really - just getting confirmation that no one sees a specific action that I missed, then if @techalchemy doesn’t respond in some time or another maintainer doesn’t remove the label then the bot will close the issue.

I would put those questions to the packaging category on Discourse (and/or the distutils-sig mailing list/pypa/packaging-problems repo) and drop a link here so interested parties can provide feedback. They are a bit higher-level than pip itself, and those resources would help expose the discussion to a wider audience who can help define best practices which could make their way into the Packaging User Guide for example.

chrahunt on Dec 8, 2019

pip needs to prepare each source requirement (run its setup.py, note the warning about PEP 517 not being supported) to get its install requirements.

But in this case, there’s a circular dependencies in the build/install requirements chain: ptyprocess list flit as a build requirement, but flit depends on requests_download, and requests_download list flit as build requirements too!

Note that this works fine if wheel installs are allowed at least for requests_download: pip download -v --no-binary :all: --only-binary requests_download ptyprocess.

benoit-pierre on Aug 25, 2018

This is a problem for Homebrew recipes of Python projects which use virtualenv_install_with_resources - this specifies --no-binary :all:.

adamtheturtle on Oct 4, 2018