MONAI: Unexpected segmentation fault encountered in worker
Describe the bug
when running the unit tests in pytorch 23.03, it sometimes exits into errors:
[2023-04-12T17:08:04.837Z] .ERROR: Unexpected segmentation fault encountered in worker.
[2023-04-12T17:08:05.092Z] ./runtests.sh: line 653: 2477 Segmentation fault (core dumped) ${cmdPrefix}${cmd} ./tests/runner.py -p "^(?!test_integration).*(?<!_dist)$"
script returned exit code 139
e.g. https://github.com/Project-MONAI/MONAI/actions/runs/4686850278/jobs/8305411337
FYI @mingxin-zheng @Nic-Ma, I haven’t found the root cause yet.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 18 (18 by maintainers)
seems to be a problem of OOM when number of threads is large and can be addressed by
OMP_NUM_THREADS=4 MKL_NUM_THREADS=4, closing this for now.