pipenv: Pipenv lock still takes way too long to complete

Why pipenv why so slow? Why?

just move to poetry - faster locks overall better performance

Same here, deleting lock file doesn’t help

just move to poetry

That’s not really constructive at all. It’s better to work on helping people here. I only suggested it as a temporary solution so people don’t have to deal with the hassle while they wait for a fix.

You wouldn’t walk into Google’s headquarters and tell everyone to “just use DuckDuckGo”, would you? It’s basic respect.

same here

The biggest problem IMO is this spinner: It’s not really informative, one cannot tell exactly what’s exactly going on right now. What if I have network problems and Pipenv experiences request timeouts? I wouldn’t know, because all progress is hidden behind spinner.

same here, there are any news?

I removed all other dependencies except the editable one. It turns out no significant difference is seen in the time costs of 2018.11.26 and 2020.8.13. It will be great if you can narrow down which package brings the performance degrading.

I created a demo repo to showcase the issue, which you can clone and adjust to reproduce the performance issue: https://github.com/frostming/pipenv-lock-performance

$ docker build . --build-arg PIPENV_VER=2020.8.13 -t pipenv-bug
$ docker build . --build-arg PIPENV_VER=2018.11.26 -t pipenv-bug

This issue is still present in April of 2021. I have yet to look into why this is, but I know it doesn’t have to be this way because there are other managers that work better and faster.

For instance, I recommend using Poetry until this is fixed. I know it seems counter-intuitive and disrespectful to recommend another project, but as of the present moment, the speed of pipenv makes it unusable, and I’d rather put an alternative on the table than leave people to wait.

I’m having this problem in February 2021. Even deleting lock file, it is still very slow.

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

Yes, +1! Thanks for looking into this, Martijn! We had exactly the same issue (lots of big tmp directories), but sorta gave up for the time being… I couldn’t really argue much when pipenv lock took 10+ minutes for simply adding another package.

(and hi, btw 👋)

I’ll be running tests later. I think the best way to resolve this is to time each stage of the lock and see which ones take the longest so we can determine where the problem comes from and attempt to fix it.

Sorry, I meant to get back to this but current project work is keeping me over-utilised. I do still find Pipenv to be excruciatingly slow, and regularly have to delete the lock file just to get a solution, but that’s not really very actionable feedback.

A “workaround” that may work (and what we have used from time to time) is to remove the lock file and rerun the lock. This may also hint at the underlying problem, which may have something to do with how pipenv is using the existing lock file during the lock process.

I’ve found the way to profile pipenv install

Marvellous. Thank you.

I’ll investigate later - although, at a glance, the presence of time.sleep is interesting.

If you have nothing constructive to say, it’s better to stay quite.

No need to be rude. I appreciated the humour honestly - we’re engineers, not lawyers, we’re allowed to be funny. You’d love some of the original UNIX design documents, developers have a real history of messing around.

Anyways, yes, I’ll be reporting the timings here in seconds, not lightyears.

requirements.txt found, instead of Pipfile! Converting...
✘ Failed... 
Warning: Your Pipfile now contains pinned versions, if your
requirements.txt did. 
We recommend updating your Pipfile to specify the "*" version, instead.
Pipfile.lock not found, creating...
start - 7.152557373046875e-07
keep outdated - 8.58306884765625e-05
cleanup lockfile - 0.0009396076202392578
inherit dev - 0.009583711624145508
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
resolve deps - 13.933054447174072
Updated Pipfile.lock (ac73fc)!
overwrite dev - 13.93443512916565
Installing dependencies from Pipfile.lock (ac73fc)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 38/38 — 00:00:15

For some strange reason it isn’t as slow on my machine anymore, only around 30 seconds. However, the bulk of the time is most definitely spent resolving and installing dependencies (in my testing, anyone else should feel free to verify this on a machine where performance is still poor) - and I’m not sure how easy it’ll be to improve that.

Thanks for raising this @mjpieters, thought I was going crazy. Downgrading to 2018.11.26 speeds things up massively for me too

anecdotally, switching my medium sized project to poetry solved all of my performance issues.

To all the people thinking that Poetry is the answer and is perfectly fast in all cases: You are wrong. Both Pipenv and Poetry are having these issues partly due to the nature how dependency management works / does not work in Python.

I did not investigate the code, but I think it happens because on every run Pipenv checks for updates even though no one asks for this, thus it makes a lot of queries, calculates a lot of hashes for Pipfile.lock etc. So the bigger your .venv is the worse things get. For example right now I installed one package, but instead of just installing it and it’s dependency Pipenv also updated three packages. As I already had 101 packages installed you can imagine the process was quite time consuming.

I have implemented Pipenv with my team recently, but the time it take to install even one package is crazy, I tried with panda, it took between 5mn to 8mn.

41 packages takes around 20mn to install.

I have tried the following :

• Deleting my virtuelenv
• Deleting Pipfile.lock

Nothing work so far

I’ve found the way to profile pipenv install, take a look at this repo for instructions. The result is *.calltree file which can be opened kcachegrind. I totally do not understand how to interpret this results, but I’m sure someone can figure it out.

Version 2020.11.15, does not take half an hour for me, but couple of minutes is too much for creating 40 KiB JSON file.

pipenv spawns a number of subprocesses during the lock, does this take any of that into account?

** edit ** I have found this is also why a lot of the output is hidden during verbose running; the stdout/stderr of the worker processes are not propagated up.

How can I run Pipenv with timings?

You have to modify the do_lock function in core.py in the root package directory to print a time.time() between calls.

It’s a bit of a hacky solution, but as far as I’m aware pipenv has no built in profiling.

but lightyears faster.

When we report the timing in our verbose output, we should use units of time; lightyears are a unit of distance (as are parsecs Han Solo)

I bet it’ll be network

I do highly doubt that. Other package management software for other ecosystems do the same thing, but lightyears faster. Unless pipenv uses a dinosaur of a networking system, it seems unlikely.

if output would be more verbose we would pinpoint problem quicker The output alone isn’t useful - we need to know the timing. We do already have a sample of verbose output, though, from earlier on in this issue.