scipy: scipy.linalg.sqrtm get stuck in SLURM clsuters

My issue is about the scipy.linalg.sqrtm getting stuck in SLURM cluster. I do not know why, when I run scipy.linalg.sqrtm in the cluster, the program will get stuck in running. But It runs well on my personal computer

Reproducing code example:

    covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False,blocksize=32)
    sigma1 and sigma2 are two 2048*2048 matrix

Error message:

    the procedure will get stuck at "  covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False,blocksize=32)"

Scipy/Numpy/Python version information:

 scipy     1.7.0+computecanada
numpy       1.20.1
python 3.7
 So please help me! thank you!

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (5 by maintainers)

Most upvoted comments

Hi, all!@ilayn @ev-br @czgdp1807 @brandon-beacher Thanks for your help! Now, I solved this problem. By setting ‘export MKL_NUM_THREADS=1’, my codes can run well. It seems there is some weird interaction between torch and scipy.

I met same error when using torch FID(mseitzer/pytorch-fid). All cpus (96 cpus) are used when using this function, and then the process stucked.

I have a server with 4 gpus, and I ran 4 programs on each gpu on parallel. Each program will compute FID. I found that all cpus will be occupied when running multi process with FID computing in the same time, but one process ran well. I think there might be some threads conflicts between torch, scipy.linalg.sqrtm (in FID) and multi-process/threads.

Setting “export MKL_NUM_THREADS=1” didn’t help. But I found that setting “export OPENBLAS_NUM_THREADS=1” or “export OMP_NUM_THREADS=1” does help for me. So anyone who has the same issue can give a try.

Related Issue: https://github.com/numpy/numpy/issues/8120#issuecomment-252058015 Related Blog: pythonspeed.com/articles/concurrency-control