pip: New resolver: Failure when package only specified with extras is not available from index

Sorry for the delay. The issue happens when pip sees a direct URL requirement, and then a requirement of the same name with different extras. Since a requirment with different extras represent a different set of dependencies, the resolver treats these two as different requirements, and is not able to resolve the non-URL requirement into using that URL.

The workaround to this would be to install the editable/URL requirements first, like this:

$ pip install --use-feature=2020-resolver -r requirements.txt --no-deps
$ pip install --use-feature=2020-resolver -r requirements.txt

The first --no-deps installation would skip all the resolver stuff (thus avoid the error). Once all those packages are in site-packages, subsequent install calls would work correctly.

Some additional technical context that might make more sense to those into pip internals: Since a[x] provides a different set of packages from a, and that set may be different depending on the version of a chosen, the resolver internally creates a “fake” package that depends on the dependencies specified by extra x, and a that matches the same version of itself. This makes pip aware of the correct version of a when it sees a[x]. But this does not work the other way around—if you provide a first, the resolver does not know it needs a[x] when it sees a (it can’t just pull it in, otherwise the user would get unwanted packages when they don’t want a[x] but only a), and we don’t have a good way to inform the resolver about that direct URL can also be used to satisfy a[x] later on. This is why installing the packages without dependencies would work around the issue. The on-disk .dist-info can be used to represent that a package, so we don’t need to use the URL.

Ultimately, this is one of those issues that are probably fixable, but require too much resource that are more useful spent elsewhere. I would be happy to offer advices if anyone really wants to work on this, but am not personally very motivated to do the work myself.

I have merged #8939, #9143, and #9204 here, which all have the same root cause, described above (https://github.com/pypa/pip/issues/8785#issuecomment-678885871).

Just been wondering if there’s been any updates to this, since 20.3/new resolver will definitely break us in various ways unless this is fixed.

Your issue is not related to this issue, and I would suggest

1. Open a new issue (I think there is already one open on this but can’t find it, so might as well start a new conversation).
2. Provide clear steps to reproduce your issue. It is close to impossible to know what is going on when all you provide are some private (and fake) repository URLs because you can have literally anything inside it, and those things are causing problems.

Hi @uranusjr , I can see https://github.com/pypa/pip/pull/9775 is merged and therefore I’m using pip of version 21.1. A similar error is raised when I try to install the following:

git+ssh://git@github.com/my-organization/repo1.git@1.0.0#egg=repo1[extra]
git+ssh://git@github.com/my-organization/repo2.git@2.0.0#egg=repo2

When repo2 depends on repo1 with no extras (same version):

git+ssh://git@github.com/my-organization/repo1.git@1.0.0#egg=repo1

The exception is:

ERROR: Cannot install -r requirements.txt (line 3) and repo1[extra]==1.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    repo2 2.0.0 depends on repo1 1.0.0 (from git+ssh://****@github.com/my-organization/repo1.git@1.0.0#egg=repo1)
    repo1[extra] 1.0.0 depends on repo1 1.0.0 (from git+ssh://****@github.com/my-organization/repo1.git@1.0.0#egg=repo1[extra])

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Could you please help me understand what I’m doing wrong?

@delijati The reason to your error is not the same as described in this issue. The two lau URLs are different. Yes, the two URLs ultimately download the same code, but pip does not have any special knowledge about GitHub, and therefore (correctly) treated as different packages.