scipy: scipy.linalg.sqrtm get stuck in SLURM clsuters
My issue is about the scipy.linalg.sqrtm getting stuck in SLURM cluster. I do not know why, when I run scipy.linalg.sqrtm in the cluster, the program will get stuck in running. But It runs well on my personal computer
Reproducing code example:
covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False,blocksize=32)
sigma1 and sigma2 are two 2048*2048 matrix
Error message:
the procedure will get stuck at " covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False,blocksize=32)"
Scipy/Numpy/Python version information:
scipy 1.7.0+computecanada
numpy 1.20.1
python 3.7
So please help me! thank you!
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 17 (5 by maintainers)
Hi, all!@ilayn @ev-br @czgdp1807 @brandon-beacher Thanks for your help! Now, I solved this problem. By setting ‘export MKL_NUM_THREADS=1’, my codes can run well. It seems there is some weird interaction between torch and scipy.
I have a server with 4 gpus, and I ran 4 programs on each gpu on parallel. Each program will compute FID. I found that all cpus will be occupied when running multi process with FID computing in the same time, but one process ran well. I think there might be some threads conflicts between torch, scipy.linalg.sqrtm (in FID) and multi-process/threads.
Setting “export MKL_NUM_THREADS=1” didn’t help. But I found that setting “export OPENBLAS_NUM_THREADS=1” or “export OMP_NUM_THREADS=1” does help for me. So anyone who has the same issue can give a try.
Related Issue: https://github.com/numpy/numpy/issues/8120#issuecomment-252058015 Related Blog: pythonspeed.com/articles/concurrency-control