pip: New Resolver unable to handle multiple versions of git dependencies

@aaraney @andylamp @goodspark it isn’t just the protocol or differences in the repo branch, it looks like ANY difference in the URL will trigger this. Confirmed by trying each of the following variants, simplified to aaraney’s example: B’s setup.py

install_requires=[
"A@ git+https://github.com/someuser/A.git",
]

Then to install both from requirements.txt (or in any other way where dependencies are resolved, I would imagine) You would have to write the requirement verbatim as defined in the install_requires

git+https://github.com/someuser/A.git
git+git://github.com/someuser/B

Even a simple change in the url causes an unresolvable dependency, i.e ommit the .git or just change case of part of the url

git+https://github.com/someuser/A
git+git://github.com/someuser/B

git+https://github.com/SOMEUSER/A.git
git+git://github.com/someuser/B

git+https://github.com/SOMEUSER/A.git#egg=thing
git+git://github.com/someuser/B

You can install A using any of the above variants by itself, but not in conjunction with B which defines install_requires

Then you specify the URL in service and apps that don’t depend on service?

But it sounds to me like you have a more complicated project dependencies than the URL is designed to handle. I would strongly encouraged switching to a setup with internal (i.e. non-PyPI) package index, and specify the index to fetch commoncode etc. with --index-url instead.

If you ask me, URL specifications are the significant problem going forward, so there’s no surprise here 🙂

There’s no way to know whether different URLs point to the logical same thing without actually downloading and potentially building the content, which is very problematic in more than one way. So systems using URLs for dependency specification generally impose restrictions over its usage. Go (for which URLs are one of the two primary ways to specify dependencies), for example, behave the same as pip here.

The legacy resolver was way too loose, relying on the user doing the right thing (only the user knows foo @ FOO and foo @ foo point to the same thing), but then things blow up too badly when the user inevitably does something wrong. The new resolver tries to bring things to where they should, at the expense of packages not “playing nice” with the newly enforced (but IMO reasonable) rules. Eventually we do want to offer a way for users to get around the issue and have a way to still use those problematic packages (#8076), but first we need to actually put up the rules to get an idea what we need to allow users to do.

@hellkite500 I think that’s the problem I am hitting - I am using two packages A and B that depend on the same package C; C resides both in the requirements.txt of A and B as well as in their install_requires.

If the URL strings aren’t exact in all places then most likely. The quick workaround is making then match I think. Not a good sustainable solution though IMHO.

@andylamp in your case it’s probably because one URL explicitly mentions the @main branch and the other not.

The egg= format cannot be used in setup.py. The @ format (PEP 508) is universal.

From what I can tell, all you need to do is to use the same format in both setup.py and requirements.txt?

But is the current behaviour not what “other dependencies” do? If you have

# service
commoncode==1.0.0

# app
commoncode==1.0.1
service==2.1.0

The installation is supposed to fail, because there’s no way to accept both commoncode 1.0.0 and 1.0.1.

I should have someway of describing something like a allowable range and/or explicit allows/denials.

You do, that’s what versions are for. You can specify

# service 2.1.0
commoncode~=1.0.0

# app
commoncode @ git+git://github.com/repoOrg/common.git@v1.0.1
service @ git+git://github.com/repoOrg/service.get@v2.1.0

(Or you can specify the URL in service and use version in app if that’s what you want instead.)

I described in #9229 how the URLs from dependencies must match exactly to resolve correctly, and why this is an important restriction.

My suggestion is to to use PEP 508 instead, which is functionally the same as the #egg= fragment, but better in every other ways.

@Tbilgere-td just to add to the discussion, reverting back to pip 20.2.4 everything works as expected.

From my working hypothesis, the string literal comparison would fail because, you know the @main part.

or in @andylamp 's case I think the issue is the #egg=myorg-logger and/or @main

@aaraney in my case all packages strictly use git+https to fetch them in all instances, just to provide some clarity on my end 😄