MONAI: Unexpected segmentation fault encountered in worker

Describe the bug

when running the unit tests in pytorch 23.03, it sometimes exits into errors:


[2023-04-12T17:08:04.837Z] .ERROR: Unexpected segmentation fault encountered in worker.

[2023-04-12T17:08:05.092Z] ./runtests.sh: line 653:  2477 Segmentation fault      (core dumped) ${cmdPrefix}${cmd} ./tests/runner.py -p "^(?!test_integration).*(?<!_dist)$"

script returned exit code 139

e.g. https://github.com/Project-MONAI/MONAI/actions/runs/4686850278/jobs/8305411337

FYI @mingxin-zheng @Nic-Ma, I haven’t found the root cause yet.

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 18 (18 by maintainers)

Most upvoted comments

seems to be a problem of OOM when number of threads is large and can be addressed by OMP_NUM_THREADS=4 MKL_NUM_THREADS=4, closing this for now.