Corrfunc: script hangs when using unbuffered output
General information
- Corrfunc version: 2.4.0
- platform: cluster with CentOS 7.4.1708
- installation method (pip/source/other?): first pip, then source
Issue description
i’m running Corrfunc on a simulation snapshot: i’m computing the angular correlation function in thin shells using DDtheta_mocks. I first installed Corrfunc via pip, then, in order to increase performance, via source:
$ git clone https://github.com/manodeep/Corrfunc/
$ make
$ make install
$ python -m pip install . --user
$ make tests
However, while with the pip install the code would parallelize on multiple threads, now it runs mostly on one thread. I’m submitting my SLURM job via: #!/bin/bash #SBATCH --ntasks=1 #SBATCH --nodes=1 #SBATCH --mem=480000 #SBATCH --exclusive […] export OMP_NUM_THREADS=48 srun -n 1 python -u $PY_CODE > $LOGS
Expected behavior
To run on 48 threads at ~100%.
Actual behavior
To mostly run on 1 thread. I checked with htop.
What have you tried so far?
To re-install it from source.
Minimal failing example
i’m attaching the log file corrfunc-logs.txt, including the ‘make tests’.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (1 by maintainers)
Commits related to this issue
- Add additional check to tell if it's safe to redirect stdout/err. Closes #269. — committed to manodeep/Corrfunc by lgarrison 3 years ago
hi @lgarrison, it works! Now the verbose output is printed into the slurm job’s standard error.
Regarding binutils/AVX-512, is it necessary to create e new environment? also, shouldn’t there be a
makebefore pip in #196 (comment)?