pipenv: pipenv fails to install all my packages, and then installs all of them
Hi, it might be something that I do not understand or so. I have 2 sources specified and none of them is official pypi, but mirrors, so here is my Pipfile:
[[source]]
url = "https://company/api/pypi/pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[[source]]
url = "https://company/api/pypi/pypi-local/simple"
verify_ssl = true
name = "pypi-local"
[packages]
cachetools = "==2.0.1"
click = "==6.2"
data-object = "==1.2.1"
flask-cache = "==0.13.1"
flask-classy = "==0.6.10"
flask-humanize = "==0.3.0"
flask-migrate = "==2.1.1"
flask-oauthlib = "==0.9.4"
flask-sqlalchemy = "==2.3.2"
flask-wtf = "==0.14.2"
flask = "==0.12.2"
huey = "==1.6.0"
pendulum = "==1.4.0"
pyyaml = "==3.12"
redis = "==2.10.6"
requests = "==2.18.4"
requests-cache = "==0.4.13"
sqlalchemy = "==1.2.1"
whitenoise = "==3.3.1"
gunicorn = "==19.7.1"
[dev-packages]
ipdb = "*"
ipython = "*"
flask-shell-ipython = "*"
coverage = "*"
flake8 = "*"
flask-debugtoolbar = "*"
Describe your environment
- OS Type Ubuntu
- Python version:
$ python -V
Python 3.6.1 - Pipenv version:
$ pipenv --version
pipenv, version 9.0.3
Expected result
All packages installed immediatly.
Actual result
Installing dependencies from Pipfile.lock (7e9249)…
An error occurred while installing blinker==1.4! Will try again.
An error occurred while installing click==6.2! Will try again.
An error occurred while installing coverage==4.4.2! Will try again.
An error occurred while installing decorator==4.2.1! Will try again.
An error occurred while installing flask-debugtoolbar==0.10.1! Will try again.
An error occurred while installing flask-shell-ipython==0.3.0! Will try again.
An error occurred while installing ipdb==0.10.3! Will try again.
An error occurred while installing ipython==6.2.1! Will try again.
An error occurred while installing ipython-genutils==0.2.0! Will try again.
An error occurred while installing jedi==0.11.1! Will try again.
An error occurred while installing parso==0.1.1! Will try again.
An error occurred while installing pexpect==4.3.1; sys_platform != 'win32'! Will try again.
An error occurred while installing pickleshare==0.7.4! Will try again.
An error occurred while installing prompt-toolkit==1.0.15! Will try again.
An error occurred while installing ptyprocess==0.5.2! Will try again.
An error occurred while installing pygments==2.2.0! Will try again.
An error occurred while installing simplegeneric==0.8.1! Will try again.
An error occurred while installing traitlets==4.3.2! Will try again.
An error occurred while installing wcwidth==0.1.7! Will try again.
Installing initially–failed dependencies…
Success installing blinker==1.4!
Success installing click==6.7!
Success installing coverage==4.4.2!
Success installing decorator==4.2.1!
Success installing flask-debugtoolbar==0.10.1!
Success installing flask-shell-ipython==0.3.0!
Success installing ipdb==0.10.3!
Success installing ipython==6.2.1!
Success installing ipython-genutils==0.2.0!
Success installing jedi==0.11.1!
Success installing parso==0.1.1!
Success installing pexpect==4.3.1; sys_platform != 'win32'!
Success installing pickleshare==0.7.4!
Success installing prompt-toolkit==1.0.15!
Success installing ptyprocess==0.5.2!
Success installing pygments==2.2.0!
Success installing simplegeneric==0.8.1!
Success installing traitlets==4.3.2!
Success installing wcwidth==0.1.7!
Steps to replicate
pipenv install --system Please also note, that pipenv is installing click==6.7 despite specifying 6.2 version.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 78
- Comments: 65 (18 by maintainers)
Commits related to this issue
- Pin pipenv version to 2018.10.13 pipenv upstream issue: https://github.com/pypa/pipenv/issues/1356 — committed to inspectorioinc/docker-python by thuandt 5 years ago
- Makefile: add `--sequential` to pipenv install command There some sort of conflicts related to the concurrent installation: https://github.com/pypa/pipenv/issues/1356#issuecomment-470847880 — committed to emlid/ntripbrowser by yashin-alexander 5 years ago
- Makefile: add `--sequential` to pipenv install command There some sort of conflicts related to the concurrent installation: https://github.com/pypa/pipenv/issues/1356#issuecomment-470847880 — committed to emlid/ntripbrowser by yashin-alexander 5 years ago
- Makefile: add `--sequential` to pipenv install command There some sort of conflicts related to the concurrent installation: https://github.com/pypa/pipenv/issues/1356#issuecomment-470847880 — committed to emlid/ntripbrowser by yashin-alexander 5 years ago
- Try to fix pipenv failed install by using sequential Try to implement fix as seen on pypa/pipenv#1356 — committed to sanjacob/BlackboardSync by sanjacob a year ago
closing since they are successful in the end — we’ll need much more information in order to replicate
Hi @techalchemy, thank you and you’re right, it’s due to some sort of conflicts related to the concurrent installation. However, making those dependencies editable didn’t solve the issue for me.
For people TL;DR, just add
--sequential
to your installation command, such as one of the following lines:So PLEASE tell me, what context you would like to get. Because a doctor doesn’t simply stand there, waiting for the patient to tell everything without asking questions.
PLEASE PROVIDE CONTEXT BEFORE PASTING OUTPUT INTO AN OPEN ISSUE.
We know what the symptom looks like. It only helps us if you provide the context that can help reproduce it. Think of it like having a dog that is sick. The issue roughly says ‘my dog is vomiting, it happens when we go on walks!’ And a bunch of others now yelling ‘my dog is also vomiting!’, ‘my cat threw up once!’, ‘my turtle threw up yesterday!’. Did they each eat bad food? What food? Were they eating the same food? If I get a turtle myself, can I copy you and get mine to vomit? Was there some other factor in play?
Hopefully this analogy makes sense. Please include details about the error and not just a dump showing that the error has occurred.
I’m having the same issue. Pretty annoying and should be looked into.
everyone, i don’t work on the pipenv project, but let me say on behalf of the maintainers: yes, it’s useful to know when a bug affects more people. it’s not useful to receive nagging comments saying “has this been fixed yet?”.
if it’s really bothering you, the right thing to do is fix it yourself. @nickwilliams-eventbrite, if you are bumping into this problem literally thousands of times like you say, it sounds like it would be a good investment on your part put some time into fixing it yourself, and saving your team time and frustration.
sorry for getting a bit preachy! but open source lives and dies on goodwill, and it’s eroded a tiny bit at a time everyone complains with a “why hasn’t this been fixed yet?” at a team of unpaid volunteers… and if this comment rubs you up the wrong way, remember to blame me, and not the lovely pipenv team 😉
I fix it with:
I have the same issue with latest version of pipenv. I’m curious, @jacekjab are you using Nexus for your repositories?
seems to be related…
pipenv install psycopg2 --sequential
worked for me!Any idea when the fix for this will release to PyPi? So far this year, over 1000 times developers in our organization have posted in our environment channel, “My container failed to build. The error is, ‘An error occurred while installing ____.’” Except that’s not the error, the error was somewhere else, and we have spent an inordinate amount of time getting people to find the actual error in the logs. This is a HUGE PAIN for us. 😦
I’m having the same issue on my CI machines (gitlab runner). I believe it’s happening due to some race condition but I can’t really nail it down.
@pawelswiecki please provide details including the output and Pipfile you used per the issue template. Telling us to look into something with no additional detail is not productive and as volunteers would rank pretty high on the list of things we also find annoying
so the issue here is just that you are installing a bunch of VCS dependencies without using
editable = true
– many thanks @hjwp for the reproduction caseI believe I have a fix for this, although for the specific case you provided there will always be a single failure that will have to get pushed off till the end due to an editable dependency that needs to be installed after another dependency (you can avoid this with
--sequential
or by making the failing dependency editable)Could reproduce this behaviour with one single installation:
It’s very likely to be caused by the psycopy2 vs psycopg2-binary mess: psycopg2-binary: Why?
The psycopg2 installation gives a warning message and is likely to cause pipenv to “fail”.
Apologies, but it is essentially impossible for me to ask you for all of the possible variables that make your environment different from our CI and the vast majority of people who use pipenv. Are you all using exclusively private repositories? Have you tried changing anything? With any success? Between which versions of pipenv did this begin occurring? I know it seems like I should have some answers but I am completely blind to the things that make your environments unique.
No. It fails to install about half of our dependencies and then comes back and says “Installing initially-failed dependencies” before finally succeeding. That’s not as expected. It is, however, as described in this bug.
Note that we don’t have any VCS dependencies. Not a one.
@changyuheng, when I add
--sequential
to my build, most of my dependencies don’t install. There’s no failure, it just skips a bunch of the dependencies inPipfile.lock
and doesn’t install them at all with no output (even in--verbose
) explaining why they were skipped. So much for that…@mcauto your issue is different, you are compiling a crypto library from source using python 3.4 (which will be end of life in just a few months).
Simply seeing an installation failure does not make two issues the same, please take the time to look at the similarities before jumping in and if your issue looks different, open another please!
Like others, I had this exact same issue. My root cause may not be the same as others, but thought I’d throw it in here to help those in a similar situation.
I’m using an old early 2011 MacBook Pro running macOS High Sierra (10.13.6). Python version is 3.7.4.
TL;DR - My build environment was out of whack. For me, the fix was reinstalling Xcode Command Line Tools with
xcode-select --install
My debugging steps:
regex
In particular, this line stuck out:
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
It turns out that I had no bin directory at all in the above path. A quick Google search revealed links such as this one https://apple.stackexchange.com/questions/321858/xcrun-missing-installing-xcode-command-line-tools-not-fixing-issue, which others basically attribute to a bad install of Xcode Command Line Tools.
After reading more from above, I realized I’d recently upgraded from an older version of macOS. When I did this, my Xcode Command Line Tools installation was no longer valid. Following the advice found in the above forum, I installed the appropriate CLT with
xcode-select --install
. After that, everything started working immediately.For me, this issue would’ve only been triggered by packages which have to be compiled.
This may or may not be similar to what others are experiencing. If so, maybe more verbose warning / error messaging from pipenv could alert the user to the root cause.
Hope this helps others.
pipenv shell
and thenpip install --upgrade setuptools
(for python 3.7.4) solved for meWe also have private repo and same issue. I’ll try to isolate the issue with a dockerfile for you, good idea!
Le dim. 7 oct. 2018 à 09:33, Dan Ryan notifications@github.com a écrit :
if you still got the error please try using this
python3 -m ensurepip --upgrade
orpy -m ensurepip --upgrade
It will upgrade the pip, and also fix my problem
@techalchemy I have a Dockerfile which reproduces this problem, or perhaps a worse one.
https://gist.github.com/hjwp/72489330407b85d7c52d29ec77c587e0
In our case pipenv reports some failures on first install, then reports attempts to reinstall, and then actually fails to install some packages altogether (but does not report an explicit error about this)
you can verify this with
We’re seeing the same thing in our jenkins builds. This output is typical:
We’re using the
python:3.6.5-jessie
docker image as our base, and we install pipenv and dependencies like this:RUN set -ex && pip install pipenv && pipenv install --system
.EDIT: Actually, we were using
python:3
in our Jenkins build. Just changed that topython:3.6.5-jessie
and we see the same behaviour: Fails installs first, then retries and succeeds.This stackoverflow response seems to solve the issue… it did for me!
I have found that this problem can also present itself if you made changes to your dependencies in your Pipfile but forgot to “lock” those changes to update the Pipfile.lock file accordingly (behaviour tested and observed in 2022.9.24, 2022.10.4, and 2022.10.25). In our specific situation, we had an internal package from an internal repo using PyPi as an upstream whose dependency declaration had been updated to a higher version in the Pipfile, but the lockfile (Pipfile.lock) still specified an older version. When running “pipenv sync” in our CI pipeline, whatever issue having this discrepancy caused prevented it from succeeding. Once we updated the lockfile accordingly and pushed the change, it resolved the issue. Hope it helps.
While installing the below lxml I am getting this error. pipenv install lxml
ERROR: Exception: Traceback (most recent call last): File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_vendor\urllib3\response.py”, line 425, in _error_catcher yield File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_vendor\urllib3\response.py”, line 507, in read data = self._fp.read(amt) if not fp_closed else b"" File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_vendor\cachecontrol\filewrapper.py”, line 62, in read data = self.__fp.read(amt) File “c:\users\shivam\appdata\local\programs\python\python38-32\lib\http\client.py”, line 458, in read n = self.readinto(b) File “c:\users\shivam\appdata\local\programs\python\python38-32\lib\http\client.py”, line 502, in readinto n = self.fp.readinto(b) File “c:\users\shivam\appdata\local\programs\python\python38-32\lib\socket.py”, line 669, in readinto return self._sock.recv_into(b) File “c:\users\shivam\appdata\local\programs\python\python38-32\lib\ssl.py”, line 1241, in recv_into return self.read(nbytes, buffer) File “c:\users\shivam\appdata\local\programs\python\python38-32\lib\ssl.py”, line 1099, in read return self._sslobj.read(len, buffer) socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\cli\base_command.py”, line 188, in _main status = self.run(options, args) File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\cli\req_command.py”, line 185, in wrapper return func(self, options, args) File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\commands\install.py”, line 332, in run requirement_set = resolver.resolve( File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\resolution\legacy\resolver.py”, line 179, in resolve discovered_reqs.extend(self._resolve_one(requirement_set, req)) File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\resolution\legacy\resolver.py”, line 362, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\resolution\legacy\resolver.py”, line 314, in _get_abstract_dist_for abstract_dist = self.preparer.prepare_linked_requirement(req) File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\operations\prepare.py”, line 467, in prepare_linked_requirement local_file = unpack_url( File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\operations\prepare.py”, line 255, in unpack_url file = get_http_url( File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\operations\prepare.py”, line 129, in get_http_url from_path, content_type = _download_http_url( File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\operations\prepare.py”, line 281, in _download_http_url for chunk in download.chunks: File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\cli\progress_bars.py”, line 166, in iter for x in it: File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_internal\network\utils.py”, line 15, in response_chunks for chunk in response.raw.stream( File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_vendor\urllib3\response.py”, line 564, in stream data = self.read(amt=amt, decode_content=decode_content) File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_vendor\urllib3\response.py”, line 529, in read raise IncompleteRead(self._fp_bytes_read, self.length_remaining) File “c:\users\shivam\appdata\local\programs\python\python38-32\lib\contextlib.py”, line 131, in exit self.gen.throw(type, value, traceback) File “C:\Users\Shivam.virtualenvs\Projects-Muah_USQ\lib\site-packages\pip_vendor\urllib3\response.py”, line 430, in _error_catcher raise ReadTimeoutError(self._pool, None, “Read timed out.”) pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=‘files.pythonhosted.org’, port=443): Read timed out.
I too have a similar issue. I tried to install
ipython
like this.pipenv install ipython
this is the output
Can someone confirm if this is still an issue?
@wjlow I think what @techalchemy meant is you need to upgrade the pip inside the tox-managed environment, not the one used by your project, i.e. add it to the
commands
config, like this:Hi @jacekjab sorry you are running into this – the initial failure isn’t actually a bug, per se, I’m guessing it is related to concurrent installs from a non-pypi repository. They initially failed and then fell back to sequential installation. As far as the click installation, you will need to provide the full context of the command you ran, your
$PATH
, the output ofenv
, that stuff would help