spack: Spack Environments are painfully slow
I’ve been using Spack Environments for a while now, and so far my biggest complaint is the performance. On an NFS cluster in particular, it can take upwards of 15 minutes just to concretize an environment. Things are particularly bad when the environment is partially installed, as most of the time is spent updating the environment, not actually concretizing anything. I’ve even noticed that commands like spack edit or spack help will take orders of magnitude longer if I’m running them from a directory containing a spack.yaml. If I run spack -d concretize, I notice that we are acquiring and releasing locks on Spack’s database thousands of times. Is there any way we can hold on to the lock until another process requests it so we don’t need all this overhead?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 8
- Comments: 24 (24 by maintainers)
Commits related to this issue
- Speedup environment activation refers #13554 — committed to alalazo/spack by alalazo 7 years ago
- Speedup environment activation refers #13554 — committed to spack/spack by alalazo 7 years ago
- Speedup environment activation refers #13554 — committed to alalazo/spack by alalazo 7 years ago
- Speedup environment activation refers #13554 — committed to alalazo/spack by alalazo 7 years ago
- Speedup environment activation refers #13554 — committed to alalazo/spack by alalazo 7 years ago
@tgamblin @adamjstewart I’m about to commit a fix that takes the 27 sec. posted above down to:
Not sure. I usually find
python -m pyinstrument $(which spack) ...useful…@alalazo @adamjstewart: just woke up or I’d do it myself, but try a quick fix by putting a read transaction around the loops in
add_default_view_to_spec(asenvironment_modifications_for_speccallsspec.prefix). This is similar to what I did in #12686.On my personal laptop, the difference is even greater. From my home directory:
From a directory containing a smaller
spack.yaml(uninstalled environment):From a directory containing a larger
spack.yaml(installed environment):That’s a 2x slow down for a small
spack.yaml, and a 30x slow down for a largespack.yamlthat has been installed.I ran some quick benchmarks on Blue Waters. From my home directory:
From a directory containing a
spack.yamlfile:At these speeds, Spack is practically useless. I can’t even
spack edita file unless I want to wait around for 3 minutes.@Sinan81 I’m guessing this is related to #13249. You can try the commits before and after if you want to confirm and give some benchmarks.