spack: Installation issue: rocblas@4.2.0%gcc@10.3.0 fails with "OSError: [Errno 12] Cannot allocate memory"

Steps to reproduce the issue

@arjun-raj-kuppala @haampie @srekolam

Trying to install ROCm 4.2.0 inside a Spack environment on a 24 core server with 64GiB RAM, 16 GiB /tmp.

spack.yaml

spack:
  concretization: separately
  packages:
    all:
      compiler: [gcc@10.3.0]
  specs:
  - matrix:
    - ['%gcc@10.3.0^cmake@3.17.5']
    - [atmi@4.2.0, comgr@4.2.0, hip@4.2.0, hipblas@4.2.0, hipcub@4.2.0, hipfft@4.2.0,
      hipfort@4.2.0, hipify-clang@4.2.0, hip-rocclr@4.2.0, hipsparse@4.2.0, hsakmt-roct@4.2.0,
      hsa-rocr-dev@4.2.0, llvm-amdgpu@4.2.0, migraphx@4.2.0, miopengemm@4.2.0, miopen-hip@4.2.0,
      miopen-opencl@4.2.0, mivisionx@4.2.0, rccl@4.2.0, rdc@4.2.0, rocalution@4.2.0,
      rocblas@4.2.0, rocfft@4.2.0, rocm-bandwidth-test@4.2.0, rocm-clang-ocl@4.2.0,
      rocm-cmake@4.2.0, rocm-dbgapi@4.2.0, rocm-debug-agent@4.2.0, rocm-gdb@4.2.0,
      rocminfo@4.2.0, rocm-opencl@4.2.0, rocm-openmp-extras@4.2.0, rocm-smi-lib@4.2.0,
      rocm-tensile@4.2.0, rocm-validation-suite@4.2.0, rocprim@4.2.0, rocprofiler-dev@4.2.0,
      rocrand@4.2.0, rocsolver@4.2.0, rocsparse@4.2.0, rocthrust@4.2.0, roctracer-dev@4.2.0] # rocm-device-libs@4.2.0,
  view: true

results in

$ spack install -j20
[…]
==> Installing rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok                                                                                         [37/152]
==> No binary for rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok found: installing from source
==> Warning: Suspicious requests to set or unset 'CXX' found
==> Warning:            env.set('CXX', os.path.join(link_dir, compiler.link_paths['cxx'])) at /sw/vulcan-CentOS7/hlrs/non-spack/misc/spack.2021-06-16/spack.de
velop/lib/spack/spack/build_environment.py:257
==> Warning:    --->     at /sw/vulcan-CentOS7/hlrs/non-spack/misc/spack.2021-06-16/spack.develop/var/spack/repos/builtin/packages/rocblas/package.py:82
==> Using cached archive: /tmp/spack/hpcoft30/src/_source-cache/archive/54/547f6d5d38a41786839f01c5bfa46ffe9937b389193a8891f251e276a1a47fb0.tar.gz
==> Using cached archive: /tmp/spack/hpcoft30/src/_source-cache/git//ROCmSoftwarePlatform/Tensile.git/3438af228dc812768b20a068b0285122f327fa5b.tar.gz
==> Warning: Fetching from mirror without a checksum!
  This package is normally checked out from a version control system, but it has been archived on a spack mirror.  This means we cannot know a checksum for th
e tarball in advance. Be sure that your connection to this mirror is secure!
==> Moving resource stage
        source : /tmp/spack/hpcoft30/build/resource-Tensile-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-src/
        destination : /tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-src/Tensile
==> No patches needed for rocblas
==> rocblas: Executing phase: 'cmake'
==> rocblas: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j20'

3 errors found in build log:
     1008        return Popen(process_obj)
     1009      File "/opt/hlrs/spack/2021-09-30/python/3.8.11-gcc-10.3.0-3abeedd2/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
     1010        self._launch(process_obj)
     1011      File "/opt/hlrs/spack/2021-09-30/python/3.8.11-gcc-10.3.0-3abeedd2/lib/python3.8/multiprocessing/popen_fork.py", line 70, in _launch
     1012        self.pid = os.fork()
     1013    OSError: [Errno 12] Cannot allocate memory
  >> 1014    make[2]: *** [Tensile/library/TensileLibrary.dat] Error 1
     1015    make[2]: Leaving directory `/tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-build-tcrmmyc'
  >> 1016    make[1]: *** [library/src/CMakeFiles/TENSILE_LIBRARY_TARGET.dir/all] Error 2
     1017    make[1]: Leaving directory `/tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-build-tcrmmyc'
  >> 1018    make: *** [all] Error 2

See build log for details:
  /tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-build-out.txt

==> Warning: Skipping build of hipblas-4.2.0-igpxkkncnsb6fk64i7qq3c2sihzpwbgf since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of miopen-hip-4.2.0-fcrcxdxgevk5x2tpulmdsyf6yclxkfx4 since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of migraphx-4.2.0-lpj3nxlbodkdys4s2ttnc64v5yrkkwb3 since miopen-hip-4.2.0-fcrcxdxgevk5x2tpulmdsyf6yclxkfx4 failed
==> Warning: Skipping build of miopen-hip-4.2.0-yqcjyia2y4z2rnshyqzqcfa6cmwardzi since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of rocsolver-4.2.0-uod4w6pjo7vezpficidsa75wcctqvuxb since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of rocm-validation-suite-4.2.0-hr2a7hn6rtyd7dz2g2quz7yqohbrttv6 since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of rocalution-4.2.0-z6kjsfpahpsjdqi4pqoin7x6bg2dn42s since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Error: Terminating after first install failure: ProcessError: Command exited with status 2:
    'make' '-j20'

Information on your system

$ spack debug report
* **Spack:** 0.16.3-4541-9e70fc6
* **Python:** 3.6.8
* **Platform:** linux-centos7-x86_64
* **Concretizer:** original

spack-build-out.txt

spack-build-env.txt

Additional information

No response

General information

  • I have run spack debug report and reported the version of Spack/Python/Platform
  • I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
  • I have uploaded the build log and environment files
  • I have searched the issues of this repo and believe this is not a duplicate

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 25 (21 by maintainers)

Most upvoted comments

Ensure you don’t have HIPCC_COMPILE_FLAGS_APPEND env variable include -parallel-jobs=X where X is > 1 set it to one, it should be defaulting to one if the flag isn’t there (this is parallel computation of a single unit but forks memory). Otherwise rocblas with Tensile will require the full 64GB RAM for building by itself so can you set your spack build to only build rocblas by itself and resume parallel package building after rocblas is done?

If you are parallel building I would drop the ratio vm.overcommit_ratio = 5 ( you don’t really want to swap), remove vm.overcommit_kbytes as you are using the ratio to see if it can still build rocblas by itself. And could also test build all packages without parallelism, it maybe you will build faster if you were trying to build rocblas with others you were swapping memory even when you succeed. Anyway cat your /proc/meminfo to ensure those look expected, and ulimit -a doens’t show memory limit.