spack: Spack Environments are painfully slow

I’ve been using Spack Environments for a while now, and so far my biggest complaint is the performance. On an NFS cluster in particular, it can take upwards of 15 minutes just to concretize an environment. Things are particularly bad when the environment is partially installed, as most of the time is spent updating the environment, not actually concretizing anything. I’ve even noticed that commands like spack edit or spack help will take orders of magnitude longer if I’m running them from a directory containing a spack.yaml. If I run spack -d concretize, I notice that we are acquiring and releasing locks on Spack’s database thousands of times. Is there any way we can hold on to the lock until another process requests it so we don’t need all this overhead?

@tgamblin

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 8
  • Comments: 24 (24 by maintainers)

Commits related to this issue

Most upvoted comments

@tgamblin @adamjstewart I’m about to commit a fix that takes the 27 sec. posted above down to:

$ time spack env activate -d --sh . &> /dev/null

real    0m3,378s
user    0m3,105s
sys     0m0,263s

Not sure. I usually find python -m pyinstrument $(which spack) ... useful…

@alalazo @adamjstewart: just woke up or I’d do it myself, but try a quick fix by putting a read transaction around the loops in add_default_view_to_spec (as environment_modifications_for_spec calls spec.prefix). This is similar to what I did in #12686.

On my personal laptop, the difference is even greater. From my home directory:

$ time spack help
...
real	0m4.016s
user	0m0.738s
sys	0m0.379s

From a directory containing a smaller spack.yaml (uninstalled environment):

$ time spack help
...
real	0m9.092s
user	0m7.952s
sys	0m0.434s

From a directory containing a larger spack.yaml (installed environment):

$ time spack help
...
real	2m7.786s
user	2m1.647s
sys	0m2.491s

That’s a 2x slow down for a small spack.yaml, and a 30x slow down for a large spack.yaml that has been installed.

I ran some quick benchmarks on Blue Waters. From my home directory:

$ time spack help
...
real	0m22.465s
user	0m2.236s
sys	0m1.344s

From a directory containing a spack.yaml file:

$ time spack help
...
real	2m58.860s
user	2m2.912s
sys	0m15.233s

At these speeds, Spack is practically useless. I can’t even spack edit a file unless I want to wait around for 3 minutes.

@Sinan81 I’m guessing this is related to #13249. You can try the commits before and after if you want to confirm and give some benchmarks.