pip: pip unexpectedly not installing latest version of git package with branch/commit pinning

In my opinion, this is not a good resolution of this ticket. This behaviour breaks the rule of least astonishment, because it goes against specificity. If this were CSS, the version number in setup.py would be a .class, and the tag referenced in requirements.txt would be an #id. The version number is set by the maintainer and applies to everyone that uses the package, the tag in requirements.txt is set by the user and applies only to this environment. It is much more specific.

Can you give me an example scenario where a user might target a specific commit and actually want to just keep the package they already have?

I think the software should respect the specificity in the requirements.txt file. If that’s not feasible, it should at least not fail silently and should alert the user that their requested branch/tag/commit was not respected.

FWIW, I struggled for 4 hours yesterday with a package and neither bumping the version number in my targetted branch, nor passing the -U flag worked. I still haven’t got my preferred version installed even now.

Here is my fab script that sets up the environment:

def setup_venv(build):
    with cd('/webapps/{0}'.format(APP_NAME)):
        if exists('venv'):
            sudo('rm -R venv')
        sudo('virtualenv venv')
        for key, value in ENV_VARS[build.lower()].items():
            var_string = 'export {0}="{1}"'.format(key, value)
            append('venv/bin/activate', var_string, use_sudo=True)
        with prefix('source venv/bin/activate'):
            sudo('pip install -U pip')
            sudo('pip install -U -r requirements.txt -r prod-requirements.txt')

and here is the requirement I can’t seem to install properly:

git+git://github.com/skolsuper/pybbm.git@tf-master#egg=pybb

I seem to end up with just the master branch every time. (In case you check the repo, I already tried bumping the version to 999, and rolled back that commit when it didn’t work)

@dstufft I am raising this issue to your attention since you seem involved in the #2432 conversation regarding shallow Git clones as well as P/Rs (now closed) #453/#507 that are possibly relevant for a solution to this issue. Today I was burned by updating a Git checkout in requirements.txt like so:

# old:
git+git://github.com/celery/celery@d7d4c2a40ee4f482ae043f92e2a5391d395027a7#egg=celery[redis]
# new:
git+git://github.com/celery/celery@120770929f4a37c5373a378b75b5c41a99702af9#egg=celery[redis]

As discussed here, this will not re-fetch the new Git source code. The work-around stated above to pass the editable flag does work:

-e git+git://github.com/celery/celery@120770929f4a37c5373a378b75b5c41a99702af9#egg=celery[redis]

Of course… now I have a 54MB tree (.git directory) instead of a smaller 6MB tree (which will be duplicated in all my succeeding Docker images). I’d be happy to pass some other flag like -s git+git://... for a shallow-clone, but, failing that, perhaps PR #507 makes sense to know if source should be re-fetched (to resolve this issue)?

BTW, if the latest code in the checked out source has a greater version number, will the new code be installed even if no --upgrade flag was specified? Since the requirement of the package being installed would be already met.

Yes, that is correct.

Even if the checked out source has a greater version number, the code will not be installed unless there is the --upgrade flag. Actually it seems that even when the version number didn’t change, it is still installed.

Ok, I (more or less) understand why pip works that way, but I have to say it really breaks expectations in a major way and that the proposed solution (using editable installs) is a workaround at best, with obviously quite a few downsides, cf the previous comments. Not such a problem in my case ATM (the few packages I use with commits as version are small) but if someone can think of possible solution to really fix this issue I’ll be happy to give a hand.

@skolsuper The only downside I’m aware of is that you have the entire git repository still on your system whereas with noneditable installs you just end up with the source. So it’s not maximally lean but I’d say that cost is generally negligible. Hopefully we’ll cut it down even further eventually with shallow clones.