spack: Staging dirs disappear for failed builds.

I build a long list of packages regularly via a Jenkins job.

When a package fails to build, its staging directory is often removed, much to my frustration.

Here’s what I see in Jenkins log file (the last line from spack is the ...'make'... line, the it goes on with other output from the driver script):

[...]
==> Installing mono
==> Fetching file:///tmp/apps-build-mirror.DiDlFd/mirror/mono/mono-5.0.1.1.tar.bz2
==> Staging archive: /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd/mono-5.0.1.1.tar.bz2
==> Created stage in /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd
==> Ran patch() for mono
==> Building mono [AutotoolsPackage]
==> Error: ProcessError: Command exited with status 2:
    'make' '-j28'
OTHER SCRIPT OUTPUT

Here’s my etc/spack/config.yaml file, I believe this says to stage things in /tmp and link them into the Spack tree:

# -------------------------------------------------------------------------
# This is the project specific spack configuration file.
# -------------------------------------------------------------------------
config:
  # Temporary locations Spack can try to use for builds.
  #
  # A value of $spack/var/spack/stage indicates that Spack should run
  # builds directly inside its install directory without staging them in
  # temporary space.
  #
  # The build stage can be purged with `spack purge --stage`.
  build_stage:
    - $tempdir
    - $spack/var/spack/stage

The link in the Spack tree exists but it points to a directory within /tmp that does not. Other Spack related staging dirs do exist there, they’re all empty. /tmp is part of the 698GB / partition, which on this machine is currently 2% full.

[hartzelg@lb097hmdev spack]$ ls -l /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd
lrwxrwxrwx 1 hartzelg users 44 Aug  4 11:31 /scratch/hartzelg/spack-cime-sc1-working-dir/workspace/sc1-daily-build/spack/var/spack/stage/mono-5.0.1.1-hx7nxp5x4irgecrvw6mwl354ehdejztd -> /tmp/hartzelg/spack-stage/spack-stage-rUCQxf
[hartzelg@lb097hmdev spack]$ ls -l /tmp/hartzelg/spack-stage/spack-stage-rUCQxf
ls: cannot access /tmp/hartzelg/spack-stage/spack-stage-rUCQxf: No such file or directory
[hartzelg@lb097hmdev spack]$ ls -l /tmp/hartzelg/spack-stage/spack-stage-* | head
/tmp/hartzelg/spack-stage/spack-stage-0ATOqP:
total 0

/tmp/hartzelg/spack-stage/spack-stage-1foqeC:
total 0

/tmp/hartzelg/spack-stage/spack-stage-3OlQZn:
total 0
[...]

Have I mis-understood something or am I tripping over a bug?

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 23 (23 by maintainers)

Most upvoted comments

This is on a single node, not a cluster (it happens to be a submit node, but it’s all running locally).

I’ve noticed (thanks to texlive being a PITA…) that builds that fail on a different system using the same scripts and etc… leave (at least they left twice) directories around for me to debug.

I’m mystified, but leaning towards it being my fault, or the platform.

I recently debugged another problem on this system, I’d unpack a tarball full of data into a “scratch” directory, set something running overnight only to discover large portions of the data missing the next morning. We dug up this little gem, which was supposed to be cleaning up files that hadn’t been accessed recently:

define(access_age,(CURRENT_TIMESTAMP - MODIFICATION_TIME))
RULE 'purge.1' DELETE WHERE((NAME LIKE '%') AND (access_age > INTERVAL '14' DAYS))

Oops.

I’m going to clone and gut the build scripts and see if I can generate an intentional test case that runs quickly. Then I’ll see what I can pin down.

Thanks to all for the followup and help …

Of course, now that I’ve added instrumentation the build didn’t fail last night…

@scheibelp – thanks for the suggestion, I’ll see if I can set that up.

[…] maybe choosing a short absolute path for build_stage in config yaml will avoid issues with overly-long paths:

If I specify a /short/path then I’ll still end up with a link from .../var/spack/stage to it, it just won’t be in /tmp. Right?

Now that I type that, I’m back to not quite understanding why just:

  # The build stage can be purged with `spack purge --stage`.
  build_stage:
    - $spack/var/spack/stage

was leading to the “argument list too long” error.

I’ll try your suggestion, and perhaps try to provoke the other problem again and take better notes.

Also, if you run spack stage package-X and the stage for package-X remains that would be useful to know.

I often run spack stage <package-foo> so that I can poke around at the innards of foo, that seems to work.

Is /tmp being purged by the system? If so you can just make $spack/var/spack/stage first in your list, and the directories won’t be purged. Otherwise, it’s a bug with Spack.