asdf-nodejs: Global installs extremely slow

It seems that doing any npm install -g is an order of magnitude slower with asdf-nodejs vs. a standard npm install.

My guess is that it is coming from the post install hook the seems to run on every dependency and every nested dependency for global installs. The post install hook seems to invoke bin/get-bin-names.js for each package, which just parses package.json to find bin files to shim. I’m guessing a big chunk of that time is spent starting up a node process for each package.

I’d suggest we either:

  • convert get-bin-names.js to a bash script that greps through the package.json to find the bin files (given the narrow acceptable values for the bin property in a package.json, grepping seems like it might be possible, or
  • convert bin/postinstall.sh to node and consolidating it with bin/get-bin-names.js, so we can avoid the hit for starting up a node process for each package

Happy to PR the latter approach (the former might stretch my bash-fu skills a bit) if there’s any interest.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 49
  • Comments: 38 (13 by maintainers)

Commits related to this issue

Most upvoted comments

Ok, so I pushed a new mechanism for auto-reshimming that doesn’t rely on running postinstall hooks on every package. Instead, the new mechanism wraps around the npm command and reshims only on global installs and uninstalls, and only after every package is installed, any other command is just a pass-through to npm.

I also added a step for removing the old postinstall script files automatically. So you guys will probably see something like this in your next npm install -g on already installed versions: “This nodejs installation is using an older auto-reshimming mechanism, we are now removing this mechanism from all nodejs installations…”. That pretty much solves the issue with slowness on installations

This new mechanism is still pretty much in testing and I would really appreciate any feedback on it. If the slowness still persists or you guys are still seeing those pesky postinstall messages, please let me know

Let me add to that: I’m not even sure this actually IS a problem, I mean I’ve used Slackware for a few years after 2001, and I’ve used Gentoo (I actually have a PowerPC that I use as a server running it in my living room), and compiling stuff takes FOREVER, and if you tell that to the Gentoo folks the first thing they’ll ask is how many HOURS.

This doesn’t even makes sense, Gentoo and Slackware are not the same as NodeJS, +6min to install a package that in base npm takes 30-60s is a HUGE difference, and in bigger packages it can take even more, asdf it’s introducing linear time to the installation with a considerable big coefficient

I don’t dare to quote your “free stuff no whining” argument because every time a read it I get a bit more salty. Just because a tool is open source doesn’t mean that issues should be ignored, this is why the issues tab exists for starters. This issue has been triaged and have space for improvement, there’s many ways we can amortize this and I’m actively researching a way to fully prevent it

I don’t get why this issue is even open I mean it’s from 2017, seriously, it’s obviously not even enough of a problem to most people…

A issue is not “closed” just because it is old or because people don’t bother with it that much, sometimes the issue is a intended behavior and won’t be “fixed”, not in this case though. A issue is a change that can be made while still applicable, there’s plenty of examples of older issues that were not fixed to this day in both Slackware and Gentoo that you defended. Your argument is deeply flawed and mostly a disconnected rant, if I were a repository moderator I would certainly flag it as off-topic or irrelevant

Could you people test using asdf’s master 0.6.4-dev and report back if the new shim executor feels faster?

No, it’s still super slow.

This is still kinda ridiculous for some packages. It takes almost 6 minutes to install the polymer-cli on my MacBook Pro.

time npm install -g polymer-cli

real	5m42.131s

I made some advancements in #197, @Stratus3D can you take a look? I think this a really big issue to be solved I don’t think it’s that big but I’m happy to see people doing PRs instead of just asking for stuff.

I’m not sure what needs to be done here, but it sounds like there is still a problem.

Let me add to that: I’m not even sure this actually IS a problem, I mean I’ve used Slackware for a few years after 2001, and I’ve used Gentoo (I actually have a PowerPC that I use as a server running it in my living room), and compiling stuff takes FOREVER, and if you tell that to the Gentoo folks the first thing they’ll ask is how many HOURS.

With ASDF the time saved is already gigantic, if anyone really needs more, AT LEAST send a PR, or do it in a fork or whatever. This thing is free, try to see if you can figure it out, have a blast! I don’t get why this issue is even open I mean it’s from 2017, seriously, it’s obviously not even enough of a problem to most people…

This has been an issue for quite a while. I personally think that auto-reshim is not really needed. Rather than having ASDF_SKIP_RESHIM, I would simply remove the auto-reshim feature altogether and add a note about reshim in the docs.

Maybe it’s a blessing in disguise and nothing should be installed globally lol

I settled for using a different env var in the file ~/.asdf/plugins/nodejs/bin/postinstall

Switched from

asdf reshim nodejs "$ASDF_INSTALL_VERSION"`

To

asdf reshim nodejs "$npm_config_node_version"

There may be a way to call ~/.asdf/plugins/nodejs/bin/postinstall through asdf, kinda like asdf-exec, but I have no idea. Would still run for every single package though.

For limiting to a single reshim something like Custom Shim Templates would allow it, but I don’t think it’s worth it. I’d either stick with postinstall or drop auto-reshimming entirely and allow people to decide if and how they want to automate it.

Not everyone who is annoyed by that opens or comments on an issue or even realizes it’s asdf’s fault this is happening in the first place, and also not everyone remembers updating their global packages regularly, unfortunately.

I happened to notice because I hadn’t used asdf before and it started happening together with me starting to use asdf. It is infuriating every time I install or update something globally.

I agree this is a problem and warrants an “issue” status. I mean, if I sell you pants but as side effect you walk 10x slower because the material is so stiff, it’s not a thing for a roadmap to be fixed some time in the future, it greatly reduces the value of the product… - Of course since this isn’t a commercial product, I cannot expect a speedy fix (and I don’t understand enough about this to create a PR myself), but I still think it should be recognized as a problem.

@CherryDT did you also update plugins asdf plugin update --all

This doesn’t even makes sense, Gentoo and Slackware are not the same as NodeJS, +6min to install a package that in base npm takes 30-60s is a HUGE difference, and in bigger packages it can take even more, asdf it’s introducing linear time to the installation with a considerable big coefficient

I don’t dare to quote your “free stuff no whining” argument because every time a read it I get a bit more salty. Just because a tool is open source doesn’t mean that issues should be ignored, this is why the issues tab exists for starters. This issue has been triaged and have space for improvement, there’s many ways we can amortize this and I’m actively researching a way to fully prevent it

Actually you’re the one I think is right in this one. I just pointed out you were the only one to actually get your hands dirty, which I think is how this should be done. And I never said “free stuff no whining” for the reason I don’t believe that. What I believe is “free stuff so if you want something try to help, don’t just expect others to do stuff for you”. And you perfectly fit on that, you’re nice, I loved to see a PR for that and I totally support your effort. Maybe I put it in a really bad way, I’m sorry for that. I was actually trying to point out how you were the only person to actually do something about it, and how we need more of that.

A issue is not “closed” just because it is old or because people don’t bother with it that much, sometimes the issue is a intended behavior and won’t be “fixed”, not in this case though. A issue is a change that can be made while still applicable, there’s plenty of examples of older issues that were not fixed to this day in both Slackware and Gentoo that you defended. Your argument is deeply flawed and mostly a disconnected rant, if I were a repository moderator I would certainly flag it as off-topic or irrelevant

Now you have a point here, and I guess many people would agree to that. I just happen to be among the people who thinks this would fit in a roadmap or something, not on an issue list, since it has been ignored for years without any major consequences whatsoever. Not that it matters, neither of us is actually the maintainer for this repo and the maintainer is the one who decides what’s left open and what gets closed. And BTW I used and loved both Slackware and Gentoo, but currently only use gentoo for one server that is a PowerPC, meaning I can’t run Arch on it. But you can probably find old issues on the Arch repo, and on many repositories of stuff I love, and even I may have old stuff on issues. I never said I followed this perfectly. I just pointed out, “if this was left alone for 4 years and people still use nevertheless, is it ACTUALLY an issue?”. You think so, and you did a PR, great, I hope you actually fix this, it would certainly be an improvement. Still, if/when you decide it’s sufficient, how much “room for improvement” would be insufficient to keep this open? What’s the criteria for closing it? That’s what I was trying to question.

And again, I absolutely support PRs for this or anything else that can help people. I just think it’s weird that forgotten stuff from years ago comes back to life in the notifications when there were obviously a bazillion npm installs during this time that nobody considered bad enough to come back here and do something about it (Until now of course), and I think it shows that this is not an issue, but something on a roadmap/backlog/whatever.

I made some advancements in #197, @Stratus3D can you take a look? I think this a really big issue to be solved

Do other standard plugins do automatic reshimming? At least the ones that I use regularly do not, but I haven’t examined all of them. If it’s non-standard behavior, especially given the unnecessary slowdown it causes with asdf-nodejs, it seems to make more sense to remove it. (Anyone using asdf quickly becomes familiar with the need to reshim after installing something that includes a binary.)

Even setting ASDF_SKIP_RESHIM=1 only helps a little bit, since it still triggers and runs the check (and prints the caveat) for every dependency.

Just added vue cli and it took over 6 minutes to install.

npm i -g @vue/cli
...
+ @vue/cli@4.2.3
added 1189 packages from 662 contributors in 385.805s

Gallons of slow post install calls 😦 image

Looking at asdf-vm/asdf#409 maybe some asdf-level solution might be a better idea, those are good suggestions in my opinion.

^ This looks like a promising direction, some sort of single-run postinstall reshim implementation.

Initially I thought maybe the newer npm stopped exposing the $npm_config_node_version environment variable used in d18287b041980a41dc9942b156db2fdc4101cd3a but that doesn’t seem to be the case, I added some logging to the postinstall hook to verify it’s still there.

However I noticed the hook is now being run for every dependency, whereas with an older npm I can see it’s only run a few times. Maybe it was previously being run for only those dependencies that had their own postinstall scripts? With polymer-cli as an example, npm 5.6.0 runs the script ~5 times whereas npm 6.1.0 runs it at least hundreds of times.

The documentation for scripts says:

If you want to run a specific script at a specific lifecycle event for ALL packages, then you can use a hook script. Place an executable file at node_modules/.hooks/{eventname}, and it’ll get run for all packages when they are going through that point in the package lifecycle for any packages installed in that root. Hook scripts are run exactly the same way as package.json scripts. That is, they are in a separate child process, with the env described above.

This does support my theory that the reshim hook is now being run for every single package installed during that npm install call, and the previous behaviour where the hook script was only being run a few times (maybe just for those dependencies that already had their own postinstall scripts) might’ve been a bug in npm. However I couldn’t spot any changelog entries that would support this.

The postinstall script might not be the best place to reshim since it’s called multiple times, and asdf-nodejs is mainly concerned about when the npm install command has finished. Adding some logic to the npm shim script might work better for knowing when installation has really finished, but it’s not a very reliable method for knowing which npm calls install packages.

Maybe some kind of hybrid approach would be a better option, for instance a postinstall hook could create a flag (such as an empty marker file under ~/.asdf/plugins/nodejs) after a package is installed and the npm shim could then check that flag after the real npm exits and run a reshim once before exiting itself…

In my opinion dropping auto-reshimming might be a valid option too in case this gets too difficult, but at the cost of ease of use.

Looking at https://github.com/asdf-vm/asdf/issues/409 maybe some asdf-level solution might be a better idea, those are good suggestions in my opinion.

I’m not sure what needs to be done here, but it sounds like there is still a problem.

^ I only started noticing this after updating to Node 10.14.1 from 9.11.2, using the latest version of asdf and asdf-nodejs. Not quite sure what changed. Rolling back to 9.x makes the installs fast again.

EDIT: I believe something changed between npm 5.6.0 and 6.1.0 that caused this to happen, Node 10.2.1 is fast but 10.3.0 is slow. I scoured through changelogs but didn’t spot anything suspicious…

Some more insight into the problem: running asdf reshim nodejs without a version is a lot slower than giving it a version. It seems that the ASDF_INSTALL_VERSION environment variable isn’t populated when the postinstall script is run on my machine, so it runs asdf reshim nodejs without a version number even though this clearly isn’t the intended behaviour. I don’t know nearly enough about asdf to know why this env var might be blank, so I don’t think I can attempt to fix this.

I just encountered this problem. The code seems to have moved on since the original post - bin/get-bin-names.js is no longer run in the postinstall script, only asdf reshim, but this, too, is slow to run, which adds up to a lot of time when installing packages with a lot of dependencies.

As a blunt workaround, I commented out the line in the postinstall script that runs asdf reshim - i.e. in ~/.asdf/plugins/nodejs/bin/postinstall:

#!/usr/bin/env bash

# asdf reshim nodejs "$ASDF_INSTALL_VERSION"

I then ran asdf reshim nodejs <version> once, manually, after I was finished installing. I don’t know if this approach has any unintended side effects, but it seemed to work for me, and might be useful to anyone else who has this issue right now and needs to get around it.

Sorry for the delay here. The postinstall script indeed runs for every nested dependency, making install very slow. Right now, the postinstall hook simply runs asdf reshim nodejs, but running this for every single dependency seems like a lot of overhead.
@davewasmer Can we manage to avoid starting up a new process for every single dependency? Thank you for your help.