spack: Locking problems

This is a fairly tricky problem to track down and I’m not particularly certain in what the actual problem is, so take this description with a grain of salt.

Our CI regularly shows errors of this kind:

+ spack install --source --only dependencies --show-log-on-error -j4 <spec>
==> Error: 

This is because either this or this assertion in lib/spack/llnl/util/lock.py being triggered (spack -d prints the stack trace to these lines). We suspect that we time out here due to the slow HDD.

We think the origin of this problem is the finally statement here (for brevity here the copy of the function)

    @contextlib.contextmanager
    def prefix_read_lock(self, spec):
        prefix_lock = self.prefix_lock(spec)
        try:
            prefix_lock.acquire_read(60)
            yield self
        finally:
            prefix_lock.release_read()

If prefix_lock.acquire_read times out, this will not do prefix_lock._reads += 1 and the prefix_lock.release_read() will run into assert prefix_lock._reads > 0. As I currently understand it, the code above seems wrong and I think the simplest solution would be to move the prefix_lock.acquire_read(60) out of the try: block. But I have no idea what kind of repercussions this would have (or even if our reasoning is solid).

Any kind of feedback would be greatly appreciated, I’m certainly out of my depth here @tgamblin @scheibelp

About this issue

  • Original URL
  • State: closed
  • Created 8 years ago
  • Comments: 19 (18 by maintainers)

Commits related to this issue

Most upvoted comments

I’ll be able to look into this by Monday 9/10

@healther for me it happens with uninstall, install, and reindex.

at least load and find still work 😃

I am running older code:

[root@laplace spack]# git log | head
commit 43114c2e06517c99dcab9e5c7dae16a0a8cb6a80
Author: scheibelp <scheibel1@llnl.gov>
Date:   Wed May 30 11:07:13 2018 -0700

    more-flexible user-specified dependency constraints (#8162)
    
    * allow user to constrain dependencies that are added conditionally
    
    * remove check for not-visited deps from normalize, move it to concretize. The check now runs after the concretization loop completes (so an error is only reported if the user-mentioned spec doesnt appear anywhere in the dag)