spack: spack spec is slow

$ time spack spec py-jupyter-notebook %gcc@5.4.0               
Input spec
--------------------------------
py-jupyter-notebook%gcc@5.4.0

Normalized
--------------------------------
py-jupyter-notebook%gcc@5.4.0
    ^npm
        ^node-js
            ^libtool
                ^m4@1.4.6:
            ^pkg-config
            ^python@2.7:2.7.99
                ^bzip2
                ^ncurses
                ^openssl
                    ^perl@5.14.0:
                    ^zlib
                ^readline
                ^sqlite
    ^py-ipykernel
        ^py-ipython@4.0:
            ^py-backports-shutil-get-terminal-size
                ^py-setuptools
            ^py-pathlib2
            ^py-pickleshare
            ^py-pygments
            ^py-simplegeneric
        ^py-jupyter-client
            ^py-jupyter-core
                ^py-traitlets@4.1.0:
                    ^py-decorator
                    ^py-enum34
                    ^py-ipython-genutils
            ^py-zmq@13:
                ^py-cffi
                    ^libffi
                    ^py-pycparser
                ^py-cython@0.16:
                ^py-py
                ^zeromq
                    ^libsodium
        ^py-pexpect
            ^py-ptyprocess
        ^py-tornado@4.0:
            ^py-backports-abc@0.4:
            ^py-backports-ssl-match-hostname
            ^py-certifi
            ^py-singledispatch
                ^py-six@1.9.0:
    ^py-jinja2
        ^py-babel@0.8:
            ^py-pytz
        ^py-markupsafe
    ^py-jupyter-console
        ^py-prompt-toolkit@1.0.0:1.999.999
            ^py-wcwidth
    ^py-nbconvert
        ^py-entrypoints
            ^py-pip
        ^py-mistune
        ^py-nbformat
            ^py-jsonschema
                ^py-functools32
                ^py-vcversioner
        ^py-pycurl
            ^curl@7.19.0:

Concretized
--------------------------------
py-jupyter-notebook@4.2.3%gcc@5.4.0~terminal arch=linux-ubuntu16-x86_64 
    ^npm@3.10.9%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^node-js@7.1.0%gcc@5.4.0~debug~doc~icu4c+openssl+zlib arch=linux-ubuntu1
6-x86_64 
            ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^m4@1.4.18%gcc@5.4.0+sigsegv arch=linux-ubuntu16-x86_64 
                    ^libsigsegv@2.10%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^openssl@1.0.2k%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^perl@5.24.1%gcc@5.4.0+cpanm arch=linux-ubuntu16-x86_64 
                ^zlib@1.2.11%gcc@5.4.0+pic+shared arch=linux-ubuntu16-x86_64 
            ^pkg-config@0.29.1%gcc@5.4.0+internal_glib arch=linux-ubuntu16-x86_6
4 
            ^python@2.7.13%gcc@5.4.0~tk~ucs4 arch=linux-ubuntu16-x86_64 
                ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16-x86_64 
                ^ncurses@6.0%gcc@5.4.0~symlinks arch=linux-ubuntu16-x86_64 
                ^readline@6.3%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^sqlite@3.8.5%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
    ^py-ipykernel@4.5.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-ipython@5.1.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-backports-shutil-get-terminal-size@1.0.0%gcc@5.4.0 arch=linux-ub
untu16-x86_64 
                ^py-setuptools@34.2.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^py-appdirs@1.4.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^py-packaging@16.8%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                        ^py-pyparsing@2.1.10%gcc@5.4.0 arch=linux-ubuntu16-x86_6
4 
                        ^py-six@1.10.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-pathlib2@2.1.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-pickleshare@0.7.4%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-pygments@2.1.3%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-simplegeneric@0.8.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-jupyter-client@4.4.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-jupyter-core@4.2.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^py-traitlets@4.3.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^py-decorator@4.0.9%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^py-enum34@1.1.6%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^py-ipython-genutils@0.1.0%gcc@5.4.0 arch=linux-ubuntu16-x86
_64 
            ^py-zmq@16.0.2%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^py-cffi@1.1.2%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^libffi@3.2.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^py-pycparser@2.13%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^py-cython@0.25.2%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^py-py@1.4.31%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^zeromq@4.1.4%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                    ^libsodium@1.0.11%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-pexpect@4.2.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-ptyprocess@0.5.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-tornado@4.4.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-backports-abc@0.4%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-backports-ssl-match-hostname@3.5.0.1%gcc@5.4.0 arch=linux-ubuntu
16-x86_64 
            ^py-certifi@2016.02.28%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-singledispatch@3.4.0.3%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
    ^py-jinja2@2.8%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-babel@2.3.4%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-pytz@2016.6.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-markupsafe@0.23%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
    ^py-jupyter-console@5.0.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-prompt-toolkit@1.0.9%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-wcwidth@0.1.7%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
    ^py-nbconvert@4.2.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-entrypoints@0.2.2%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-pip@9.0.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-mistune@0.7.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-nbformat@4.1.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^py-jsonschema@2.5.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^py-functools32@3.2.3-2%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
                ^py-vcversioner@2.16.0.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
        ^py-pycurl@7.43.0%gcc@5.4.0 arch=linux-ubuntu16-x86_64 
            ^curl@7.52.1%gcc@5.4.0 arch=linux-ubuntu16-x86_64 


real    0m33.437s
user    0m33.080s
sys     0m0.344s

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 21 (21 by maintainers)

Most upvoted comments

I agree with @scheibelp, if the worst known package in Spack (r-rminer) only takes 8 seconds, that’s pretty damn good. Building takes so much longer that it really doesn’t matter.

I think the only source of annoyance is that people will run spack spec before spack install, so they have to wait for the concretizer twice. Perhaps caching would help on that front.

P.S. If anyone is bored, they can package CONDOP, which depends on r-rminer. I would be really interested to see how much worse it becomes (CONDOP depends on a lot of other big bundles).

It got a lot faster, I am now getting:

$ time spack spec py-jupyter-notebook%gcc@5.4.0
...
real    0m2.327s
user    0m2.212s
sys     0m0.104s
$ time spack spec r-rminer
...
real    0m8.059s
user    0m7.852s
sys     0m0.184s

Still slow, but usable, while previously it was not actually usable.

@certik Welcome to Spack-land. I am not familiar with Hashdist, but I see you have at least 2 years experience with it. I would be interested in your answer to @adamjstewart’s question. I believe that concretization is one of Spack’s big strengths.

I agree, slow concretization is a problem; and to be honest, concretization is now 3x faster than it was 1 year ago. But as you point out, that’s not fast enough. It’s almost never a problem with C/Fortran code; sometimes a problem with Python; and as you point out, R and JavaScript involve so many packages the system quickly becomes unusable.

Unfortunately, slow concretization is not an easy problem to solve. On a theoretical basis, the problem is known to be NP-complete, but suitably solved by SAT solvers in practice. The current concretization algorithm is somewhat ad-hoc, but there are plans to replace it with a more formalized SAT solver. That could certainly yield improvements.

The point where we need to be careful, however, is what SAT solver is used. Currently, Spack is in pure Python, making it easy to download-n-go. But that could be a real impediment to fast concretization going forward. The other alternative would be to interface to some sort of compiled SAT solver. This would require compilation/installation before Spack is fully usable; a chiken-and-egg problem. A hybrid approach would be to use a “basic” pure Python solver, which is good enough to help Spack build its own compiled SAT solver.

On an operational level, it might be easier to approach this problem by reducing the need to re-concretize. If you keep saying spack install on the same spec, there is no need to keep re-running the concretization algorithm on that spec. Putting such a cache/memoization into Spack would not be too hard. Upcoming Spack Environments — which more formally separate concretization from building — might also be helpful here.

As always… PRs are welcome. One to cache concretizations would not be too hard.

@tgamblin I see, great. I just wanted to point out that from my point of view, 8s is most definitely not “pretty damn good”, but just barely usable. But we probably all have different use cases and different perspectives.

@certik: it would still help but actually I think a Python DPLL or CDCL implementation would speed things up too. Most of the time is spent in graph traversal because the Spec is rebuilt each time a constraint (like an optional dependency) is evaluated. I plan on getting rid of that and just solving on the constraints, then building the DAG. So stay tuned.