spack: Installation issue: rocblas@4.2.0%gcc@10.3.0 fails with "OSError: [Errno 12] Cannot allocate memory"
Steps to reproduce the issue
@arjun-raj-kuppala @haampie @srekolam
Trying to install ROCm 4.2.0 inside a Spack environment on a 24 core server with 64GiB RAM, 16 GiB /tmp.
spack.yaml
spack:
concretization: separately
packages:
all:
compiler: [gcc@10.3.0]
specs:
- matrix:
- ['%gcc@10.3.0^cmake@3.17.5']
- [atmi@4.2.0, comgr@4.2.0, hip@4.2.0, hipblas@4.2.0, hipcub@4.2.0, hipfft@4.2.0,
hipfort@4.2.0, hipify-clang@4.2.0, hip-rocclr@4.2.0, hipsparse@4.2.0, hsakmt-roct@4.2.0,
hsa-rocr-dev@4.2.0, llvm-amdgpu@4.2.0, migraphx@4.2.0, miopengemm@4.2.0, miopen-hip@4.2.0,
miopen-opencl@4.2.0, mivisionx@4.2.0, rccl@4.2.0, rdc@4.2.0, rocalution@4.2.0,
rocblas@4.2.0, rocfft@4.2.0, rocm-bandwidth-test@4.2.0, rocm-clang-ocl@4.2.0,
rocm-cmake@4.2.0, rocm-dbgapi@4.2.0, rocm-debug-agent@4.2.0, rocm-gdb@4.2.0,
rocminfo@4.2.0, rocm-opencl@4.2.0, rocm-openmp-extras@4.2.0, rocm-smi-lib@4.2.0,
rocm-tensile@4.2.0, rocm-validation-suite@4.2.0, rocprim@4.2.0, rocprofiler-dev@4.2.0,
rocrand@4.2.0, rocsolver@4.2.0, rocsparse@4.2.0, rocthrust@4.2.0, roctracer-dev@4.2.0] # rocm-device-libs@4.2.0,
view: true
results in
$ spack install -j20
[…]
==> Installing rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok [37/152]
==> No binary for rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok found: installing from source
==> Warning: Suspicious requests to set or unset 'CXX' found
==> Warning: env.set('CXX', os.path.join(link_dir, compiler.link_paths['cxx'])) at /sw/vulcan-CentOS7/hlrs/non-spack/misc/spack.2021-06-16/spack.de
velop/lib/spack/spack/build_environment.py:257
==> Warning: ---> at /sw/vulcan-CentOS7/hlrs/non-spack/misc/spack.2021-06-16/spack.develop/var/spack/repos/builtin/packages/rocblas/package.py:82
==> Using cached archive: /tmp/spack/hpcoft30/src/_source-cache/archive/54/547f6d5d38a41786839f01c5bfa46ffe9937b389193a8891f251e276a1a47fb0.tar.gz
==> Using cached archive: /tmp/spack/hpcoft30/src/_source-cache/git//ROCmSoftwarePlatform/Tensile.git/3438af228dc812768b20a068b0285122f327fa5b.tar.gz
==> Warning: Fetching from mirror without a checksum!
This package is normally checked out from a version control system, but it has been archived on a spack mirror. This means we cannot know a checksum for th
e tarball in advance. Be sure that your connection to this mirror is secure!
==> Moving resource stage
source : /tmp/spack/hpcoft30/build/resource-Tensile-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-src/
destination : /tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-src/Tensile
==> No patches needed for rocblas
==> rocblas: Executing phase: 'cmake'
==> rocblas: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
'make' '-j20'
3 errors found in build log:
1008 return Popen(process_obj)
1009 File "/opt/hlrs/spack/2021-09-30/python/3.8.11-gcc-10.3.0-3abeedd2/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
1010 self._launch(process_obj)
1011 File "/opt/hlrs/spack/2021-09-30/python/3.8.11-gcc-10.3.0-3abeedd2/lib/python3.8/multiprocessing/popen_fork.py", line 70, in _launch
1012 self.pid = os.fork()
1013 OSError: [Errno 12] Cannot allocate memory
>> 1014 make[2]: *** [Tensile/library/TensileLibrary.dat] Error 1
1015 make[2]: Leaving directory `/tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-build-tcrmmyc'
>> 1016 make[1]: *** [library/src/CMakeFiles/TENSILE_LIBRARY_TARGET.dir/all] Error 2
1017 make[1]: Leaving directory `/tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-build-tcrmmyc'
>> 1018 make: *** [all] Error 2
See build log for details:
/tmp/spack/hpcoft30/build/spack-stage-rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok/spack-build-out.txt
==> Warning: Skipping build of hipblas-4.2.0-igpxkkncnsb6fk64i7qq3c2sihzpwbgf since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of miopen-hip-4.2.0-fcrcxdxgevk5x2tpulmdsyf6yclxkfx4 since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of migraphx-4.2.0-lpj3nxlbodkdys4s2ttnc64v5yrkkwb3 since miopen-hip-4.2.0-fcrcxdxgevk5x2tpulmdsyf6yclxkfx4 failed
==> Warning: Skipping build of miopen-hip-4.2.0-yqcjyia2y4z2rnshyqzqcfa6cmwardzi since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of rocsolver-4.2.0-uod4w6pjo7vezpficidsa75wcctqvuxb since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of rocm-validation-suite-4.2.0-hr2a7hn6rtyd7dz2g2quz7yqohbrttv6 since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Warning: Skipping build of rocalution-4.2.0-z6kjsfpahpsjdqi4pqoin7x6bg2dn42s since rocblas-4.2.0-tcrmmycbthlflr5xqq2xfxdfpaxft3ok failed
==> Error: Terminating after first install failure: ProcessError: Command exited with status 2:
'make' '-j20'
Information on your system
$ spack debug report
* **Spack:** 0.16.3-4541-9e70fc6
* **Python:** 3.6.8
* **Platform:** linux-centos7-x86_64
* **Concretizer:** original
Additional information
No response
General information
- I have run
spack debug reportand reported the version of Spack/Python/Platform - I have run
spack maintainers <name-of-the-package>and @mentioned any maintainers - I have uploaded the build log and environment files
- I have searched the issues of this repo and believe this is not a duplicate
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 25 (21 by maintainers)
Ensure you don’t have HIPCC_COMPILE_FLAGS_APPEND env variable include -parallel-jobs=X where X is > 1 set it to one, it should be defaulting to one if the flag isn’t there (this is parallel computation of a single unit but forks memory). Otherwise rocblas with Tensile will require the full 64GB RAM for building by itself so can you set your spack build to only build rocblas by itself and resume parallel package building after rocblas is done?
If you are parallel building I would drop the ratio vm.overcommit_ratio = 5 ( you don’t really want to swap), remove vm.overcommit_kbytes as you are using the ratio to see if it can still build rocblas by itself. And could also test build all packages without parallelism, it maybe you will build faster if you were trying to build rocblas with others you were swapping memory even when you succeed. Anyway cat your /proc/meminfo to ensure those look expected, and ulimit -a doens’t show memory limit.