rustup: docs installation slow on NFS home
Problem
rustup docs installation is remarkably slow on an HPC system (probably most HPC systems). In case it’s useful to anyone, I’m observing this on HiPerGator, which uses the Lustre filesystem.
I’ve attached some strace -c output below along with what rustup printed during the update. The 433.6 KiB/s is probably mis-stating the situation; for a few seconds at a time the install speed drops to between 50 and 8.0 KiB/s as if it’s being throttled by the filesystem.
Possible Solution(s) It would be great to opt out of docs altogether; I’m told this is a feature under consideration which would be great. There’s no way for me to use the HTML docs in an HPC environment; I use the local ones on the machine I’m ssh’d from.
Notes
Output of rustup --version: rustup 1.18.3 (435397f48 2019-05-22)
Output of rustup show:
Default host: x86_64-unknown-linux-gnu
installed toolchains
--------------------
stable-x86_64-unknown-linux-gnu (default)
nightly-x86_64-unknown-linux-gnu
installed targets for active toolchain
--------------------------------------
x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl
active toolchain
----------------
stable-x86_64-unknown-linux-gnu (default)
rustc 1.34.2 (6c2484dc3 2019-05-13)
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2019-05-23, rust version 1.36.0-nightly (37ff5d388 2019-05-22)
info: downloading component 'rustc'
91.0 MiB / 91.0 MiB (100 %) 48.0 MiB/s in 2s ETA: 0s
info: downloading component 'rust-std'
61.3 MiB / 61.3 MiB (100 %) 38.4 MiB/s in 1s ETA: 0s
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: removing component 'rustc'
info: removing component 'rust-std'
info: removing component 'cargo'
info: removing component 'rust-docs'
info: installing component 'rustc'
91.0 MiB / 91.0 MiB (100 %) 11.7 MiB/s in 9s ETA: 0s
info: installing component 'rust-std'
61.3 MiB / 61.3 MiB (100 %) 10.3 MiB/s in 5s ETA: 0s
info: installing component 'cargo'
info: installing component 'rust-docs'
11.1 MiB / 11.1 MiB (100 %) 433.6 KiB/s in 1m 58s ETA: 0s
nightly-x86_64-unknown-linux-gnu updated - rustc 1.36.0-nightly (37ff5d388 2019-05-22)
strace -c output while running a rustup update nightly
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
26.12 1.577563 79 19971 unlink
22.39 1.351963 66 20379 37 open
15.56 0.939575 47 20002 fchmod
13.00 0.784812 37 21441 chmod
7.07 0.426997 3 157983 write
4.40 0.265628 13 20503 lstat
2.53 0.153032 49 3108 getdents
1.98 0.119512 5 21896 close
1.32 0.079967 396 202 read
1.26 0.076021 49 1545 openat
0.91 0.055049 5 10621 91 futex
0.83 0.050257 18 2737 510 stat
0.82 0.049569 62 803 10 mkdir
0.67 0.040695 51 794 rmdir
0.53 0.031927 1774 18 fdatasync
0.35 0.021093 1 33633 ioctl
0.17 0.009982 22 462 rename
0.05 0.002836 24 118 brk
0.02 0.001310 164 8 munmap
0.01 0.000356 27 13 copy_file_range
0.00 0.000296 148 2 clone
0.00 0.000164 6 28 mmap
0.00 0.000162 4 39 fstat
0.00 0.000124 8 15 mprotect
0.00 0.000054 27 2 1 poll
0.00 0.000037 7 5 getrandom
0.00 0.000028 5 6 rt_sigaction
0.00 0.000020 10 2 wait4
0.00 0.000018 9 2 socketpair
0.00 0.000017 4 4 2 recvfrom
0.00 0.000015 5 3 pipe2
0.00 0.000013 4 3 sigaltstack
0.00 0.000011 6 2 getrlimit
0.00 0.000007 7 1 sched_getaffinity
0.00 0.000005 1 4 lseek
0.00 0.000005 5 1 rt_sigprocmask
0.00 0.000005 5 1 fcntl
0.00 0.000005 5 1 arch_prctl
0.00 0.000005 5 1 set_tid_address
0.00 0.000005 5 1 set_robust_list
0.00 0.000004 4 1 1 rt_sigreturn
0.00 0.000004 4 1 mremap
0.00 0.000000 0 1 1 access
0.00 0.000000 0 1 execve
------ ----------- ----------- --------- --------- ----------------
100.00 6.039148 336364 653 total
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 18 (18 by maintainers)
Commits related to this issue
- Enable threaded closing on all platforms. Set RUSTUP_CLOSE_THREADS=disabled to force single threaded IO, or to a specific number if desired for testing/tuning. This may improve #1867, but has no imp... — committed to rbtcollins/rustup.rs by rbtcollins 5 years ago
- Enable threaded closing on all platforms. Set RUSTUP_CLOSE_THREADS=disabled to force single threaded IO, or to a specific number if desired for testing/tuning. This may improve #1867, but has no imp... — committed to rbtcollins/rustup.rs by rbtcollins 5 years ago
- Enable threaded closing on all platforms. Set RUSTUP_CLOSE_THREADS=disabled to force single threaded IO, or to a specific number if desired for testing/tuning. This may improve #1867, but has no imp... — committed to rbtcollins/rustup.rs by rbtcollins 5 years ago
- Enable threaded closing on all platforms. Set RUSTUP_CLOSE_THREADS=disabled to force single threaded IO, or to a specific number if desired for testing/tuning. This may improve #1867, but has no imp... — committed to rbtcollins/rustup.rs by rbtcollins 5 years ago
- Enable threaded closing on all platforms. Set RUSTUP_CLOSE_THREADS=disabled to force single threaded IO, or to a specific number if desired for testing/tuning. This may improve #1867, but has no imp... — committed to rbtcollins/rustup.rs by rbtcollins 5 years ago
Yep, so 2m38s of wallclock time, of which 16s user time (likely decompression, hashing, etc) and 6.6s of syscall CPU time. The remaining 2m14s or so will be waiting for the storage/network. 😄
As a for instance rust-docs/share/doc/rust/html/rust-by-example/fn/closures/closure_examples:
strace for closure_examples
This has a total syscall time of
Which is representative of the extra work here - we really only need to set umask appropriately, do the mkdir once, then never touch the dir again. That would halve the amount of work that the HPC file system is being asked to do.