spack: Locking problems
This is a fairly tricky problem to track down and I’m not particularly certain in what the actual problem is, so take this description with a grain of salt.
Our CI regularly shows errors of this kind:
+ spack install --source --only dependencies --show-log-on-error -j4 <spec>
==> Error:
This is because either this or this assertion in lib/spack/llnl/util/lock.py being triggered (spack -d prints the stack trace to these lines). We suspect that we time out here due to the slow HDD.
We think the origin of this problem is the finally statement here (for brevity here the copy of the function)
@contextlib.contextmanager
def prefix_read_lock(self, spec):
prefix_lock = self.prefix_lock(spec)
try:
prefix_lock.acquire_read(60)
yield self
finally:
prefix_lock.release_read()
If prefix_lock.acquire_read times out, this will not do prefix_lock._reads += 1 and the prefix_lock.release_read() will run into assert prefix_lock._reads > 0. As I currently understand it, the code above seems wrong and I think the simplest solution would be to move the prefix_lock.acquire_read(60) out of the try: block. But I have no idea what kind of repercussions this would have (or even if our reasoning is solid).
Any kind of feedback would be greatly appreciated, I’m certainly out of my depth here @tgamblin @scheibelp
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Comments: 19 (18 by maintainers)
I’ll be able to look into this by Monday 9/10
@healther for me it happens with
uninstall,install, andreindex.at least
loadandfindstill work 😃I am running older code: