rustup: Installing rust-docs component on Windows 10 is very slow

At this point I think the only fair thing to do is make rust-docs installation slow on all other platforms too. That way the predominantly linux/mac using community of Rust will finally care about this problem and get it fixed.

The sources are still 2,307 files. While not as bad as the 15,377 files of the docs, it’s still a hefty chunk of time. Really we should just not extract rust-docs until the user asks for the docs. That way people who are offline can still have the docs without remembering to install them ahead of time, without making installation and upgrading such a slow process for other users.

Really we should just not extract rust-docs until the user asks for the docs.

@retep998 This seems very reasonable, and I agree with @kryptan that most users will never use rust-docs (unless somehow some IDE makes use of it automatically). For example, I know intellij-rust can navigate into the std source from user code which is pretty nice.

What would be the workflow envisioned for this? Something like the following?

$ rustup install toolchain X
... (installs rust-docs compressed file, but does not extract)

... (some time later)
$ rustup component add rust-docs
... (this extracts the rust-docs compressed file, so works even without an Internet connection)

@kinnison: while I do agree that minimal profile allows to avoid installing documentation and resolves #998, the fact that documentation installation is slow on Windows is still valid.

Rather than fixing this, Windows should definitely realize that slowing their system down like this isnt acceptable.

It’s an issue in Windows, not rustup.

@rbtcollins Even if we had perfect disk access, installing rust-docs would still take a long time. This issue is for taking measures such as not extracting the docs until needed or even not downloading them at all.

I’m now involved in a discussion with Defender folk ; they’ve asked for traces of the poor behaviour. Please use rustup 1.18.3 or newer, nothing older, as there is no point sending in unoptimised traces IMO.

We only need 3-4 traces from a few different machines where Defender overhead is the problem; I’ll be submitting one from a surface pro and one from a 2990WX. If someone has a machine with e.g. spinning metal disks, or even less cores or whatever - something interestingly different, please add the trace id here.

For us the use case we’re tracing is unpack performance, so I’d suggest running rustup uninstall nightly, rustup install nightly waiting for it to start installation, then copy the contents of ~/.rustup/downloads/* to a temp dir. The rustup uninstall nightly. Then copy that temp dir contents back to the the downloads dir.

Finally, start the trace and then run rustup install nightly.

Instructions for gathering a trace (note this captures most metadata about what the system is doing…):

1. Open the feedback hub
2. report a problem (link back to here)
3. in the category section select problem
4. select ‘security and privacy’ as the category
5. and ‘windows defender antivirus’ as the sub-category
6. click on re-create my problem
7. start the capture
8. rustup install nightly
9. stop the capture
10. submit the feedback item
11. add the feedback id here

It is also slow if you install Rust in WSL. Filesystem is just very slow on Windows. I would prefer to not install docs if it was possible.

I remember watching some video based on this issue about io performance and ms defender. Can anyone send link here please?

@uis246 Are you looking for https://lobste.rs/s/yne5kz/ntfs_really_isn_t_bad_robert_collins_lca?

You have little hope at staying in the FOSS world for your toolchains and build systems if you are using Windows, else you’d be suffering quite big constraints like this performance one. Stuff to make cab or msi files arent FOSS. Code signing certificates are the same thing, they’re an administrative cost.

Windows Defender who is the principal contributor to this issue, is spinning up whole arbitrary file type parsers between system calls, there’s nothing to do about it but to change Windows Defender and thus Windows, who has it enabled by default. I understand the problem of using a monolithic system where any program gets access to the whole filesystem without an Anti-Virus, but it’s not like we can’t do better than monolithic these days.

As a workaround, I would be looking at trying to extract the file in a way that Windows Defender can clearly recognize the relationship between it’s archived copy and it’s extracted copy, Windows Defender will scan archives when they are written already, if it can identify that the file is being copied out of an archive it already scanned, it does not need to scan it again.

You might want to use a third party tool for this, a third party tool that is trusted by Windows Defender, that most commonly happens when it has a digital signature with some history of having good behavior. Windows might have some command line tools to extract archives.

I agree with @retep998 , #904 can improve things, but with this amount of files, I don’t think it’s a “strict duplicate”. It’s related, but not duplicate.

I think this is a strict duplicate of #904 - the rust-docs component shows the pathology most severely, but its not unique to rust-docs.

A related issue reported on the User’s Forum and #1464:

The problem I am seeing is: at the end of the updating done by Rustup it hangs. The CPU and disk usages are both zero. I don’t know what’s happening but it’s not copying files, nor scanning them. And I’ve kept the shell open for hours with no changes

Another data point from @chriskrycho:

As it stands, Windows builds on CI take ~5⨉ longer than Linux or macOS just to get through the setup step 3, because NTFS kind of chokes when dealing with a large number of small files