rustup: rustup is still slow under WSL

Problem We forgot about WSL when making the threading code OS specific.

Steps

  1. Run rustup in WSL on Windows 10 (a WSL backed to an NTFS volume)

Possible Solution(s) Perhaps thread always? Or perhaps try to detect WSL?

Notes

Output of rustup --version: Output of rustup show:

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 1
  • Comments: 15 (14 by maintainers)

Commits related to this issue

Most upvoted comments

So the model I now have is:

  • operations that change the fd<->Handle mapping will contend with each other hugely
  • those operations appear to do their external work:
    • in the lock (close)
    • before the lock (open)
  • defender will make closing files slow - don’t know yet if flushing a file will cause it to scan it or only close

Things we haven’t modelled:

  • what sort of lock is it? do multiple closes content with each other, or just close with opens?

Possible workarounds to try:

  • opening many files in parallel: even if close is serialised we can get up to hardware-thread opens per close() because close does the OS work outside the lock
  • syncing data before closing (try making close faster)
  • don’t close any files until we have hardware-thread-files-ready, then close one per hardware thread at the same time (if close doesn’t mutually exclude other closes, these can proceed in parallel)

First experiment, enabling threads under WSL was entertaining. Made no impact. Something in LXCORE.SYS appears to be serialising all IO. Hah. Hah. Hah. Probably this is one of the drivers for WSL2…

So one option is we could include the uname crate if we don’t already, and then use uname::uname()?.release.find("Microsoft") to detect WSL.

Alternatively we just enable threading for closes always – I would like us to have a more robust limit on open handles at that point, (perhaps limited queue length would be sufficient?)