gitlint: Binary .local/bin/gitlint missing after update from 0.16.0 to 0.17.0

After upgrading to the latest release 0.17.0 the executable is gone. Uninstalling and installing again does not help. When installing 0.16.0 all is fine.

After installing 0.17.0 …

$ pip3 install --user -U gitlint
Collecting gitlint
  Using cached gitlint-0.17.0-py2.py3-none-any.whl (2.7 kB)
Requirement already satisfied: gitlint-core[trusted-deps]==0.17.0 in ./.local/lib/python3.6/site-packages (from gitlint) (0.17.0)
Requirement already satisfied: arrow>=1 in ./.local/lib/python3.6/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint) (1.2.1)
Requirement already satisfied: Click>=8 in ./.local/lib/python3.6/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint) (8.0.3)
Requirement already satisfied: sh>=1.13.0 in ./.local/lib/python3.6/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint) (1.14.2)
Requirement already satisfied: python-dateutil>=2.7.0 in ./.local/lib/python3.6/site-packages (from arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (2.8.0)
Requirement already satisfied: typing-extensions in ./.local/lib/python3.6/site-packages (from arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (3.7.4.3)
Requirement already satisfied: importlib-metadata in ./.local/lib/python3.6/site-packages (from Click>=8->gitlint-core[trusted-deps]==0.17.0->gitlint) (4.8.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil>=2.7.0->arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (1.11.0)
Requirement already satisfied: zipp>=0.5 in ./.local/lib/python3.6/site-packages (from importlib-metadata->Click>=8->gitlint-core[trusted-deps]==0.17.0->gitlint) (3.6.0)
Installing collected packages: gitlint
Successfully installed gitlint-0.17.0
WARNING: You are using pip version 21.0.1; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

… no executable to be found

$ find .local -name *gitlint*
.local/lib/python3.6/site-packages/gitlint_core-0.17.0.dist-info
.local/lib/python3.6/site-packages/gitlint-0.17.0.dist-info

After installing 0.16.0 …

$ pip3 install --user -U gitlint==0.16.0
...

… everything is fine

$ find .local -name *gitlint*
.local/lib/python3.6/site-packages/gitlint_core-0.17.0.dist-info
.local/lib/python3.6/site-packages/gitlint
.local/lib/python3.6/site-packages/gitlint/files/gitlint
.local/lib/python3.6/site-packages/qa/test_gitlint.py
.local/lib/python3.6/site-packages/qa/__pycache__/test_gitlint.cpython-36.pyc
.local/lib/python3.6/site-packages/gitlint-0.16.0.dist-info
.local/bin/gitlint

$ pip3 --version
pip 21.0.1 from /usr/lib/python3.6/site-packages/pip (python 3.6)

About this issue

Original URL
State: open
Created 3 years ago
Comments: 18 (9 by maintainers)

Commits related to this issue

requirements[dev_helper]: Upgrade gitlint from >=0.10 to >=0.17. This may require extra development documentation due to jorisroovers/gitlint#250, which applies for upgrades. — committed to neiljp/zulip-terminal by neiljp 2 years ago
requirements[dev_helper]: Upgrade gitlint from >=0.10 to >=0.17. This may require extra development documentation due to jorisroovers/gitlint#250, which applies for upgrades. — committed to zulip/zulip-terminal by neiljp 2 years ago

Most upvoted comments

The solution is to do a clean gitlint install rather than an in-place upgrade: pip uninstall gitlint; pip install gitlint

More precisely, it should be pip uninstall gitlint gitlint-core; pip install gitlint which would also take care of a semi-broken install.

At this point, I think we should close this particular issue

At Ansible, we’ve opted for documenting this in the FAQ: https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_4.html#other / https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.10.html#known-issues. Maybe, gitlint could do the same? Probably, we could change the issue label to docs at this point.

webknjaz on Nov 22, 2022

@jorisroovers I wish I saw this issue earlier. I know exactly what happens. We’ve seen this when we were splitting the project under ansible/ansible that was previously published as just ansible on PyPI but got renamed to ansible-base, and then ansible-core. The ansible distribution is also published but with different contents.

TL;DR this is a limitation of pip that is described at https://github.com/pypa/pip/issues/8509.

When you rename a distribution package (the name published on PyPI), but keep that importable package (the folder with Python files that’s installed under site-packages), there are two different packages that provide the same files (or partially the same). There used to be a gitlint/ directory installed into site-packages/ in gitlint==0.16.0. Now, there’s none, but a folder with the same name gitlint/ is shipped as a part of gitlint-core which is pulled in by gitlint. When you do a clean install, pip will resolve the deptree and unpack the dists properly.

When you have gitlint==0.16.0, there’s already site-packages/gitlint/ on disk. And pip knows about each file belonging to the said package. During the dependency resolution, it knows that it needs to install gitlint==0.17.0 and gitlint-core==0.17.0 but because of one being a dependency of the other, it knows that gitlint-core should go in first. So pip goes ahead and unpacks the contents of gitlint-core into site-packages/gitlint/, overwriting the existing files. At this point, pip knows that the files belong to gitlint==0.16.0 and it needs to uninstall it before writing the newer version to disk, it removes these files which got installed from gitlint-core because the gitlint==0.16.0’s metadata still says that they belong to this package. And then, it goes ahead and installs gitlint==0.17.0. This is how you end up with a broken installation. It is specific to pip upgrades, but not fresh installs. The issue I linked above has a number of suggestions that pip could implement to address this bug (including transactions, similar to other packaging ecosystems), but so far, nobody’s working on that.

webknjaz on Nov 21, 2022

On the uninstall: I think we’ve all mentioned that gitlint-core needs to also be uninstalled for full cleanup. We can update docs to point this out or further investigate if that can be fixed in setup.py. Separate issue IMO.

jorisroovers on Dec 3, 2021

Ok, I couldn’t resist. I booted an SUSE Linux Enterprise Server 15 SP2 instance on AWS, and still can’t reproduce this, see below.

ec2-user@ip-172-31-37-40:~> cat /etc/os-release
NAME="SLES"
VERSION="15-SP2"
VERSION_ID="15.2"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp2"

ec2-user@ip-172-31-37-40:~> sudo zypper install -y python3-pip git
# removed output
ec2-user@ip-172-31-37-40:~> sudo pip3 install -U pip==21.0.1
# removed output
ec2-user@ip-172-31-37-40:~> python3 --version
Python 3.6.12
ec2-user@ip-172-31-37-40:~> pip3 --version
pip 21.0.1 from /home/ec2-user/.local/lib/python3.6/site-packages/pip (python 3.6)

ec2-user@ip-172-31-37-40:~> pip3 install --user gitlint
Collecting gitlint
  Downloading gitlint-0.17.0-py2.py3-none-any.whl (2.7 kB)
Collecting gitlint-core[trusted-deps]==0.17.0
  Downloading gitlint_core-0.17.0-py2.py3-none-any.whl (64 kB)
     |████████████████████████████████| 64 kB 6.1 MB/s
Collecting sh>=1.13.0
  Downloading sh-1.14.2-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 11.1 MB/s
Collecting Click>=8
  Downloading click-8.0.3-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 13.0 MB/s
Collecting arrow>=1
  Downloading arrow-1.2.1-py3-none-any.whl (63 kB)
     |████████████████████████████████| 63 kB 5.2 MB/s
Collecting typing-extensions
  Downloading typing_extensions-4.0.0-py3-none-any.whl (22 kB)
Requirement already satisfied: python-dateutil>=2.7.0 in /usr/lib/python3.6/site-packages (from arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (2.7.3)
Collecting importlib-metadata
  Downloading importlib_metadata-4.8.2-py3-none-any.whl (17 kB)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil>=2.7.0->arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint) (1.14.0)
Collecting zipp>=0.5
  Downloading zipp-3.6.0-py3-none-any.whl (5.3 kB)
Installing collected packages: zipp, typing-extensions, importlib-metadata, sh, Click, arrow, gitlint-core, gitlint
  WARNING: The script gitlint is installed in '/home/ec2-user/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Click-8.0.3 arrow-1.2.1 gitlint-0.17.0 gitlint-core-0.17.0 importlib-metadata-4.8.2 sh-1.14.2 typing-extensions-4.0.0 zipp-3.6.0
WARNING: You are using pip version 21.0.1; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

ec2-user@ip-172-31-37-40:~> find .local -name *gitlint*
.local/lib/python3.6/site-packages/gitlint
.local/lib/python3.6/site-packages/gitlint/files/gitlint
.local/lib/python3.6/site-packages/gitlint_core-0.17.0.dist-info
.local/lib/python3.6/site-packages/gitlint-0.17.0.dist-info
.local/bin/gitlint

ec2-user@ip-172-31-37-40:~> .local/bin/gitlint --version
gitlint, version 0.17.0

# ~/.local/bin/gitlint is NOT in $PATH, but we've already established this is not the issue

Now I’m really out of time for today 😃 Appreciate if you can provide the steps to reproduce.

jorisroovers on Nov 30, 2021

@webknjaz thanks for providing that extra context here, that’s insightful!

At this point, I think we should close this particular issue as the root cause and workaround is known, this probably only affects a (very) small fraction of our users and we can’t really solve this on our end.

To summarize:

This will only occur when upgrading from gitlint <= 0.16
The solution is to do a clean gitlint install rather than an in-place upgrade: pip uninstall gitlint; pip install gitlint

jorisroovers on Nov 22, 2022

Thanks for digging into this @omarkohl, this helped. I’ve been able to reproduce this in a virtualenv on macOS. I am now convinced this is indeed a gitlint installation bug and not specific to any OS or python/pip version.

Using the verbose flag on pip -v helped me to confirm that this is because pip removes bin/gitlint as part of the gitlint 0.16.0 uninstallation. You can easily see in the output (below), this happens after installation of gitlint and gitlint-core.

As @sigmavirus24 suggested, I also think this happens because we moved the gitlint “binary” (really the entry_point script) to gitlint-core as part of 0.17.0. From pip’s point of view, there’s a conflict there: 2 different packages are trying to own the same file.

I’m not a pip expert, but I’m guessing pip just uninstalls everything it finds in site-packages/gitlint-0.16.0.dist-info/RECORD, unless referenced from newer versions (and same file checksum or something). Since gitlint 0.17.0 installation record is not referencing bin/gitlint (again, it’s part of gitlint-core now), pip will just remove it. Maybe this is not exactly how it works, but I think this is conceptually what’s going on.

I think if we were to release 0.17.1, this problem would not occur when upgrading from 0.17.0 to 0.17.1 (or beyond). However, this problem will continue to exist for upgrading from 0.16.0 (and before). Maybe there’s ways to fix that by adding special installation steps in setup.py, but IMO we can also just add a note to the Changelog and/or pin a GH issue asking folks to do a clean re-install when upgrading from before 0.16.0 and leave it at that.

For the record, I haven’t actually tried this. Appreciate any help getting this confirmed (or new insights 😃 )

(.venv)  ✘  /tmp  python --version
Python 3.10.0
(.venv)  /tmp  pip --version
pip 21.3.1 from /private/tmp/.venv/lib/python3.10/site-packages/pip (python 3.10)

(.venv)  /tmp  pip -v install gitlint==0.16.0
Using pip 21.3.1 from /private/tmp/.venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting gitlint==0.16.0
  Using cached gitlint-0.16.0-py2.py3-none-any.whl (82 kB)
Collecting Click==8.0.1
  Using cached click-8.0.1-py3-none-any.whl (97 kB)
Collecting arrow==1.2.0
  Using cached arrow-1.2.0-py3-none-any.whl (62 kB)
Collecting sh==1.14.2
  Using cached sh-1.14.2-py2.py3-none-any.whl (40 kB)
Collecting python-dateutil>=2.7.0
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, python-dateutil, sh, Click, arrow, gitlint
  changing mode of /private/tmp/.venv/bin/gitlint to 755
Successfully installed Click-8.0.1 arrow-1.2.0 gitlint-0.16.0 python-dateutil-2.8.2 sh-1.14.2 six-1.16.0
(.venv)  /tmp  gitlint --version
gitlint, version 0.16.0
(.venv)  ✘  /tmp  find .venv -name '*gitlint*'
.venv/bin/gitlint
.venv/lib/python3.10/site-packages/gitlint-0.16.0.dist-info
.venv/lib/python3.10/site-packages/gitlint
.venv/lib/python3.10/site-packages/gitlint/files/gitlint
.venv/lib/python3.10/site-packages/qa/__pycache__/test_gitlint.cpython-310.pyc
.venv/lib/python3.10/site-packages/qa/test_gitlint.py

(.venv)  /tmp  pip -v install gitlint==0.17.0
Using pip 21.3.1 from /private/tmp/.venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting gitlint==0.17.0
  Using cached gitlint-0.17.0-py2.py3-none-any.whl (2.7 kB)
Collecting gitlint-core[trusted-deps]==0.17.0
  Using cached gitlint_core-0.17.0-py2.py3-none-any.whl (64 kB)
Requirement already satisfied: sh>=1.13.0 in ./.venv/lib/python3.10/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint==0.17.0) (1.14.2)
Requirement already satisfied: Click>=8 in ./.venv/lib/python3.10/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint==0.17.0) (8.0.1)
Requirement already satisfied: arrow>=1 in ./.venv/lib/python3.10/site-packages (from gitlint-core[trusted-deps]==0.17.0->gitlint==0.17.0) (1.2.0)
Collecting Click>=8
  Using cached click-8.0.3-py3-none-any.whl (97 kB)
Collecting arrow>=1
  Using cached arrow-1.2.1-py3-none-any.whl (63 kB)
Requirement already satisfied: python-dateutil>=2.7.0 in ./.venv/lib/python3.10/site-packages (from arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint==0.17.0) (2.8.2)
Requirement already satisfied: six>=1.5 in ./.venv/lib/python3.10/site-packages (from python-dateutil>=2.7.0->arrow>=1->gitlint-core[trusted-deps]==0.17.0->gitlint==0.17.0) (1.16.0)
Installing collected packages: Click, arrow, gitlint-core, gitlint
  Attempting uninstall: Click
    Found existing installation: click 8.0.1
    Uninstalling click-8.0.1:
      Removing file or directory /private/tmp/.venv/lib/python3.10/site-packages/click-8.0.1.dist-info/
      Removing file or directory /private/tmp/.venv/lib/python3.10/site-packages/click/
      Successfully uninstalled click-8.0.1
  Attempting uninstall: arrow
    Found existing installation: arrow 1.2.0
    Uninstalling arrow-1.2.0:
      Removing file or directory /private/tmp/.venv/lib/python3.10/site-packages/arrow-1.2.0.dist-info/
      Removing file or directory /private/tmp/.venv/lib/python3.10/site-packages/arrow/
      Successfully uninstalled arrow-1.2.0
  changing mode of /private/tmp/.venv/bin/gitlint to 755
  Attempting uninstall: gitlint
    Found existing installation: gitlint 0.16.0
    Uninstalling gitlint-0.16.0:
      Removing file or directory /private/tmp/.venv/bin/gitlint
      Removing file or directory /private/tmp/.venv/lib/python3.10/site-packages/gitlint-0.16.0.dist-info/
      Removing file or directory /private/tmp/.venv/lib/python3.10/site-packages/gitlint/
      Removing file or directory /private/tmp/.venv/lib/python3.10/site-packages/qa/
      Successfully uninstalled gitlint-0.16.0
Successfully installed Click-8.0.3 arrow-1.2.1 gitlint-0.17.0 gitlint-core-0.17.0

(.venv)  /tmp  gitlint --version
pyenv: gitlint: command not found

(.venv)  ✘  /tmp  find .venv -name '*gitlint*'
.venv/lib/python3.10/site-packages/gitlint_core-0.17.0.dist-info
.venv/lib/python3.10/site-packages/gitlint-0.17.0.dist-info

jorisroovers on Dec 3, 2021

@jorisroovers thanks for taking the time to look into it! I will try to report more data. (sidenote: I just had a look at your blog, very nice 👍)

omarkohl on Nov 30, 2021

Hmm, ok. I don’t have more time right now though, appreciate if you can help getting to a reproducible scenario, maybe on OpenSUSE (preferably using Vagrant).

I’m intrigued because we did change how we do packaging in 0.17.0, so it’s possible that affects installation for some users.

jorisroovers on Nov 30, 2021