vuepress: Vuepress building is too slow ?!

@ulivz I wouldn’t wipe this issue out so quickly. Sure @JimmyVV should provide more data but I’m migrating a documentation (5000+ files) from Metalsmith to VuePress and the main issue we have now is that we can’t deploy on Netlify because the build time is longer than the maximum build timeout. Metalsmith did build the whole doc within Netlify’s timeouts.

Ok, Vuepress’ build process is surely more complex than Metalsmith and Vuepress is indeed a better tool, but yet… The build time is incredibly bigger in VP and this is strange.

So this is meant to add some data to the “comparison with other static website building tools” you asked above, @ulivz .

@JimmyVV please, would you post your investigations about memleaks introduced by the default theme?

Another point that cannot be ignored is that when @JimmyVV downgrades, the perfs are better. I definitely think we should look into this.

Recently, I used node workers to speed up the compiling process. the total time decrease from 10min to 2min. Maybe this is another way to improve the compiling procedure, by worker thread.

Disable the plugin-last-updated, may reduce half build time. For a repo with less than 200 (dynamic generated) docs:

• turn on: 45.51s
• turn off: 23.76s

themeConfig: {
  lastUpdated: false
}

The plugin need spawn a child process to run a local git query on each file, which can be especially slower on cloud with slow disk. https://github.com/vuejs/vuepress/blob/master/packages/%40vuepress/plugin-last-updated/index.js

@JimmyVV maybe you could share your work?

First of all, why is it slow? The compilation time depends on the size of the content you build. You have 6000 files to compile, isn’t that slow?

Secondly, let’s talk about the necessity of this issue. You can try any other static website building tool to build it. If they are very fast, please use accurate data to show that slow construction is a unique issue of vuepress.

We ran this in prod okay today but I’ve got some work to be done on prepping that PR, including things I’d like to smooth over with my implementation:

1. Make the thread count and verbose flags config driven
2. Extend the worker logging to include progress bars
3. Perhaps even do some Node version sniffing to allow it to build as-is without requiring Node’s worker_threads for better backwards compatibility.

Plus things that aren’t working for me from https://github.com/vuejs/vuepress/blob/master/.github/CONTRIBUTING.md that I need to chew on:

1. yarn bootstrap is not a defined script and yarn boot throws Error: Cannot find module '@vuepress/shared-utils'
2. The commit message convention link is broken

All that said, if anyone is actively following this and curious, you can see the current state of my work at https://github.com/itsxallwater/vuepress/commit/fa4c703f9e297a9e9de2f67a8e41d7fbbd6f2a2a.

I’ve got an update to core/lib/node/build/index.js and an addition of core/lib/node/build/worker.js to leverage Node’s worker_threads that I’ve been testing out.

For background, we’ve got a repo with ~2250 pages (producing a dist of ~4800 files / 350 MB) that was building in 1 hour, 20 minutes with the current release of Vuepress. With these changes in, I’m now getting the following timings:

1x worker thread = 1 hour 2x worker threads = 26 minutes 4x worker threads = 13.5 minutes (ironically the last minute was console updates from workers on what they were rendering, the actual rendering was already done) 8x worker threads = 8.5 minutes (I added in a verbose flag to toggle the console write of the specific document name being rendered) 16x worker threads = 7.5 minutes

I’ll get a PR in for this tomorrow since it seems there could be interest but figured I’d at least leave a note here. Will reference once there’s a PR.

@bretterer if you’re so inclined, I updated my fork to 1.3 and made a few adjustments to get my worker_threads version working again. You can find it at https://github.com/itsxallwater/vuepress/tree/feat/implement-node-worker-threads

Haven’t had a chance to revisit the contribution guide to work out my issues getting Yarn and the monorepo/workspace stuff sorted out, which would lend itself towards getting this in as a PR. @kefranabg with the work you’re putting into perf improvements at the moment, is that a worthwhile thing to pursue? I am seeing additional build runtime improvements in taking advantage of node’s worker_threads.

@itsxallwater thanks a lot for your work on the worker threads, we’ll surely try it out one of these days (our doc is 6000+ .md files).

@bretterer concerning the memleak, we’ve tried to take a look at it one year ago and found out that it was not a memleak, but rather vue-ssr that loads all the documents in memory before starting the build. I’m not 100% sure of this information, it should be verified.

You can see what I am doing in order to use the workers you suggested here without us having to maintain a fork of vuepress ourselves.

https://github.com/okta/okta-developer-docs/blob/master/packages/%40okta/vuepress-site/.vuepress/scripts/build.sh

Just a gentle touch to note I’m thinking about revisiting this. Our docs continue to grow as we add more and more product and our build times (running with GitHub Actions) are creeping towards 40 minutes.

@itsxallwater I tried your fork on some internal API documentation. It’s ~4000 markdown files. Previously our build took 4-5 hours, but with your fork, it’s ~20 minutes. Thanks! I’m looking forward to seeing this in master. Your command line additions will be welcome as well.

@bretterer if you’re so inclined, I updated my fork to 1.3 and made a few adjustments to get my worker_threads version working again. You can find it at https://github.com/itsxallwater/vuepress/tree/feat/implement-node-worker-threads

@itsxallwater Ill take a look on tuesday! Thank you!

It appears that part of the build process spins up many node processes however at least with the standard build command rendering the static content only utilizes a single node process. Perhaps utilizing available cores for static content generation would speed this up greatly for people?

Thanks @itsxallwater … We were just talking about this and have not been able to leave v1.2 because of the size of our documentation. There is for sure a memory leak as sites grow bigger. builds start out fast, and then quickly get slower and slower the more pages it works through.

Can I recommend https://github.com/josdejong/workerpool for a worker threads implementation? It has some nice features that sound relevant from the discussion:

1. Node.js detection to fallback to child processes if worker threads aren’t supported
2. Automatically determines thread count based on host’s CPU if not specified

Wish I had time to contribute more to this, good luck!

While I’m working on VuePress build time improvement, using @itsxallwater might be a good woarkaround https://github.com/itsxallwater/vuepress/tree/feat/implement-node-worker-threads 👍

Just had a chance to update to 1.3 and try @bretterer and you’re correct. Given that @kefranabg has picked this up odds are his path will carry us to victory 😉