pipenv: `pipenv install XYZ` command sometimes hangs indefinitely at "Locking [packages] dependencies"
Similar to these issues:
Summary
When using pipenv install XYZ
, the process hangs indefinitely at “Locking [packages] dependencies”.
This doesn’t happen every time, and may be related to which package is being installed. This morning it is happening every time I try to install tensorflow. Last night it was happening for numpy as well, but today numpy is fine.
The hanged response seems to last indefinitely (though I did manually terminate the process after an hour). The Pipfile.lock is not written, but the Pipfile is written and the expected packages are installed and listed when I run pipenv run pip freeze
.
I was able to reproduce this using various permutations of:
- (L)ubuntu 16.04 and (X)ubuntu 17.10
pipenv --three
andpipenv --two
- pipenv installed through python 2.7 and pipenv installed through python 3.5
- …and I downloaded the pipenv source and installed that in an isolated environment to see if the issue was specific to pipenv version 11.8.3. I experienced the same behavior in 11.9.0.
I am also able to reproduce it by simply running pipenv lock
or pipenv update
if I have tensorflow (or whatever package is currently causing me trouble) listed in my Pipfile.
$ python -m pipenv.help output
Pipenv version: '11.8.3'
Pipenv location: '/home/mary/.local/lib/python2.7/site-packages/pipenv'
Python location: '/usr/bin/python'
Other Python installations in PATH
:
-
2.7
:/usr/bin/python2.7
-
2.7
:/usr/bin/python2.7
-
3.5
:/usr/bin/python3.5m
-
3.5
:/usr/bin/python3.5
-
2.7.12
:/usr/bin/python
-
2.7.12
:/usr/bin/python2
-
3.5.2
:/usr/bin/python3
PEP 508 Information:
{'implementation_name': 'cpython',
'implementation_version': '0',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '4.13.0-37-generic',
'platform_system': 'Linux',
'platform_version': '#42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018',
'python_full_version': '2.7.12',
'python_version': '2.7',
'sys_platform': 'linux2'}
System environment variables:
MANDATORY_PATH
_LXSESSION_PID
XDG_GREETER_DATA_DIR
PROJECT_HOME
LC_CTYPE
PYTHONDONTWRITEBYTECODE
XDG_CURRENT_DESKTOP
XDG_SESSION_TYPE
LOGNAME
XDG_SEAT
PATH
XDG_VTNR
QT_PLATFORM_PLUGIN
PYTHONUNBUFFERED
VIRTUALENVWRAPPER_SCRIPT
ZSH
DISPLAY
SSH_AGENT_PID
LANG
TERM
SHELL
XDG_SESSION_PATH
XAUTHORITY
LANGUAGE
SHLVL
QT_LINUX_ACCESSIBILITY_ALWAYS_ON
QT_QPA_PLATFORMTHEME
SESSION_FOLDER
QT_ACCESSIBILITY
WINDOWID
LIBVIRT_DEFAULT_URI
HOME
XDG_SESSION_DESKTOP
SAL_USE_VCLPLUGIN
XDG_RUNTIME_DIR
WORKON_HOME
SSH_AUTH_SOCK
VTE_VERSION
GDMSESSION
VIRTUALENVWRAPPER_WORKON_CD
XDG_SEAT_PATH
PIP_PYTHON_PATH
XDG_SESSION_ID
DBUS_SESSION_BUS_ADDRESS
_
VIRTUALENVWRAPPER_HOOK_DIR
VIRTUALENVWRAPPER_PROJECT_FILENAME
DESKTOP_SESSION
LSCOLORS
XDG_CONFIG_DIRS
DEFAULTS_PATH
XDG_CONFIG_HOME
OLDPWD
LS_COLORS
GDM_LANG
XDG_DATA_DIRS
PWD
XDG_MENU_PREFIX
LESS
PAGER
USER
Pipenv–specific environment variables:
Debug–specific environment variables:
PATH
:/home/mary/bin:/home/mary/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL
:/usr/bin/zsh
LANG
:en_US.UTF-8
PWD
:/home/mary/Development/Projects/poor_mans_smart_camera
Contents of Pipfile
(‘/home/mary/Development/Projects/poor_mans_smart_camera/Pipfile’):
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[dev-packages]
[packages]
numpy = "*"
tensorflow = "*"
[requires]
python_version = "3.5"
Expected result
I expect the step in which the Pipfile.lock is generated to take at most a minute.
I have good hardware with lots of free memory and processing power, so I don’t think this is a resource constraint. My internet speed is 60 Mbps, so I doubt it is a networking issue when requesting packages.
Actual result
While this screenshot doesn’t show verbose output like the file above, it does visually demonstrate what I am experiencing.
Steps to replicate
- Spin up new virtualenv using
pipenv --three
orpipenv --two
. - Confirm things are working as expected with a package you know is installing fine such as
pipenv install pylint
. - Now cause it to hang by installing a package that is not working for whatever reason such as
pipenv install tensorflow
.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 29
- Comments: 78 (32 by maintainers)
Commits related to this issue
- Locking down packages manually until pipenv matures enough Sadly due to several issues with Pipfile.lock, it seems to either hang or take forever as seen here https://github.com/pypa/pipenv/issues/22... — committed to Mathspy/tic-tac-toe-NN by Mathspy 6 years ago
- Locking down packages manually until pipenv matures enough Sadly due to several issues with Pipfile.lock, it seems to either hang or take forever as seen here https://github.com/pypa/pipenv/issues/22... — committed to Mathspy/tic-tac-toe-NN by Mathspy 6 years ago
- Locking down packages manually until pipenv matures enough Sadly due to several issues with Pipfile.lock, it seems to either hang or take forever as seen here https://github.com/pypa/pipenv/issues/22... — committed to Mathspy/tic-tac-toe-NN by Mathspy 6 years ago
Would it be feasible to display the package that’s currently being locked? I think that would at least help to stop people thinking that Pipenv is hanging.
@uranusjr @techalchemy @jtratner
Holy buckets, Batman! It worked!
Let’s see if this solution works for some of the other people reporting the same issue. @roddds @jlhood @roveo @claytonaalves If you’re still able to reproduce pipenv hanging indefinitely at “Lock… [packages] dependencies”, try clearing your pip (and pipenv?) cache at
~/.cache/.pip
and report back on whether that resolved the issue with pipenv hanging. 😃I’m currently seeing the same problem; my
pipenv lock --verbose --clear
output looks exactly like @MarenstainBear’s. I’m running Ubuntu and am using pyenv to manage virtualenvs.hardinfo_report.txt
I’m facing the same issue here. Python 3.6.4, pipenv 11.9.0. It doesn’t hang if I install with
--skip-lock
.So your prompt says
@marvel
but you’re quoting Robin…hmm 😏@techalchemy My process, as far as I can tell, is actually hanging. I’ve let it go for five hours before killing it. Looking at
top
, I didn’t see any indication that anything was actually being processed. By contrast, when I install tensorflow in my virtualenv with pip instead of pipenv, it takes just a few seconds.As you can see in this capture of my terminal, it took my system 12.25 seconds to spin up a new python3 virtualenv and use pip to install tensorflow using the command
/usr/bin/time -v pipenv run pip install tensorflow
.If I instead run the command
/usr/bin/time -v pipenv install tensorflow
, the process appears to be hanging indefinitely at the “Locking [packages] dependencies…” after the installation is already successful.You can see from this image that after about 59 seconds of actively doing something, the subprocess related to resolver.py stops actively using system resources and, as far as I can tell, will never complete. When I took this screenshot, the process had been running for 16 minutes.
Next release should help with this a bit. Should have it in a few hours.
@uranusjr I inserted some write statements into the resolver.py code, and looks to be hanging somewhere in the
pipenv.utils.resolve_deps
call at this line.In other words, resolver.py does not appear to reach its end.
@techalchemy, I ran
pipenv lock --verbose --clear
, and the results are similar:It remained hung at “Locking [packages] dependencies”.
Any luck reproducing this on your machine? With tensorflow in particular, it is happening every time for me.
EDIT: I tested this out on a friend’s computer. He’s running Ubuntu 17.10, and he did not experience the problem. Both computers are on the same network. One difference between his computer and mine is that he only installed pipenv rather than pipenv and virtualenvwrapper. I’ll spin up a virtual machine to see if this is somehow connected to virtualenvwrapper and/or the environment variables I have set that are shared across both (i.e. $WORKON_HOME).
For this particular problem,
pipenv lock --clear
should be enough.I had the same issue. Here’s where it was hanging for me:
The problem was something to do with pip trying to write something to it’s cache. I was able to resolve it by clearing my pip cache - which for Mac users is
rm -rf ~/Library/Caches/pip
.Thanks to all the contributers to this issue!
Locking simply takes a long time folks. Because of python dependency management we have to download each package and run
setup.py
to get dependencies as needed. For libraries like tensorflow and numpy you may wind up with an sdist which means you are building from source. That is slow. I’ve seen it take 10 minutes. It goes faster if you heat up the cache.I’m going to close this issue for now until someone can actually show me that their process is hung or not doing anything. We are considering ways to improve this process
Error: no such option: --clear
Same stuff here, pip env hanging on Mac OS. I’ve tried to create a simple
pipenv install flask
on a new folder only for the sake of testing. It was hanging, but I noted it’s was not using pip’s cache dir, but its own:So what worked for me was
Really strange behaviour. Found this behaviour on after upgrading python from 3.6.3 to 3.7.0
I run across the same issue with the latest Python, pip, and pipenv while installing pendulum and, unfortunately, clearing pip’s and pipenv’s cache by running `sudo rm ~/.cache/pip* -rf’ doesn’t solve the problem.
So the issue, at least for me, still persists.
lol… so clearing your pip cache fixed it… life is amusing…
Side note - It would really help if we could somehow stream output from the resolver rather than blocking until the resolver returns (just to make it easier to use verbose to debug where hang is happening) On Sun, Apr 1, 2018 at 5:21 PM MarenstainBear notifications@github.com wrote:
+1 I’m seeing the same issue trying to install numpy on MacOS. Verified I don’t see this issue installing boto3.
Yes, it appears as though I do, but only for my python 2, not for python 3 (and my pipenv instance was installed through the python 2 version of pip):
I’ll go pop over to my other laptop and into some virtual environments where I can’t reproduce this to see if they do not have pyOpenSSL.
EDIT: Deleted a couple of paragraphs about another issue I was seeing. That was my error–I was actually in a subdirectory but didn’t realize it because without my normal zsh configuration disabled for testing purposes, I no longer had a visual indicator in my prompt of my cwd. As @uranusjr correctly stated “but Pipenv does a lot of environment detection, and is known to work weirdly in an improperly-configured shell.”… I indeed had momentarily incorrectly configured my shell.
@uranusjr
EDIT 2: I tested this out on a virtual machine with a pretty similar setup to my host machine (including installing pipenv against a python 2.7.12 environment containing the pyOpenSSL module), and on that machine everything is working perfectly as expected. That gives me doubt that pyOpenSSL is the problem here. (I did note that the pipenv version is higher on my virtual machine, so I think I’ll upgrade pipenv locally to see if that makes a difference.)
$ python -m pipenv.help output (from the virtual machine)
Pipenv version:
'11.9.0'
Pipenv location:
'/home/mary/.local/lib/python2.7/site-packages/pipenv'
Python location:
'/usr/bin/python'
Other Python installations in
PATH
:2.7
:/usr/bin/python2.7
2.7
:/usr/bin/python2.7
3.5
:/usr/bin/python3.5m
3.5
:/usr/bin/python3.5
2.7.12
:/usr/bin/python
2.7.12
:/usr/bin/python2
3.5.2
:/usr/bin/python3
PEP 508 Information:
System environment variables:
XDG_GREETER_DATA_DIR
GNOME_DESKTOP_SESSION_ID
PYTHONDONTWRITEBYTECODE
LESSOPEN
XDG_SESSION_TYPE
QT_IM_MODULE
LOGNAME
USER
PROMPT_COMMAND
HOME
XDG_VTNR
PATH
PYTHONUNBUFFERED
DISPLAY
SSH_AGENT_PID
LANG
TERM
SHELL
XDG_SESSION_PATH
QT_STYLE_OVERRIDE
LANGUAGE
SESSION_MANAGER
SHLVL
QT_LINUX_ACCESSIBILITY_ALWAYS_ON
GTK_CSD
QT_ACCESSIBILITY
XMODIFIERS
GIO_LAUNCHED_DESKTOP_FILE_PID
XDG_SESSION_DESKTOP
GIO_LAUNCHED_DESKTOP_FILE
XDG_RUNTIME_DIR
SSH_AUTH_SOCK
VTE_VERSION
GDMSESSION
XDG_SEAT_PATH
LESSCLOSE
GSETTINGS_SCHEMA_DIR
XDG_CURRENT_DESKTOP
XDG_SESSION_ID
DBUS_SESSION_BUS_ADDRESS
_
XAUTHORITY
DESKTOP_SESSION
XDG_CONFIG_DIRS
GTK_MODULES
GDM_LANG
PANTHEON_TERMINAL_ID
XDG_DATA_DIRS
PWD
PIP_PYTHON_PATH
XDG_MENU_PREFIX
LS_COLORS
XDG_SEAT
Pipenv–specific environment variables:
Debug–specific environment variables:
PATH
:/home/mary/bin:/home/mary/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL
:/bin/bash
LANG
:en_US.UTF-8
PWD
:/home/mary