legion: Realm: GPUXferDes::progress_xd seg fault
I’m seeing a seg fault in S3D. Here is a stack trace:
#0 0x00001555527fb238 in nanosleep () from /lib64/libc.so.6
#1 0x00001555527fb13e in sleep () from /lib64/libc.so.6
#2 0x000015553662359e in Realm::realm_freeze (signal=11) at /lustre/scratch/vsyamaj/legion_s3d_tdb/legion/runtime/realm/runtime_impl.cc:206
#3 <signal handler called>
#4 0x00001555334f6884 in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#5 0x0000155533458043 in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#6 0x00001555335eeb91 in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#7 0x00001555338670d1 in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#8 0x00001555334aa1d6 in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#9 0x00001555334aa5f1 in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#10 0x00001555334b43a9 in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#11 0x000015553369b79b in ?? () from /cm/local/apps/cuda/libs/current/lib64/libcuda.so.1
#12 0x00001555366f86e8 in Realm::Cuda::GPUXferDes::progress_xd (this=0x154cb67b8be0, channel=0x1406e7c0, work_until=...) at /lustre/scratch/vsyamaj/legion_s3d_tdb/legion/runtime/realm/cuda/cuda_internal.cc:792
#13 0x000015553670680e in Realm::XDQueue<Realm::Cuda::GPUChannel, Realm::Cuda::GPUXferDes>::do_work (this=0x1406e7f8, work_until=...) at /lustre/scratch/vsyamaj/legion_s3d_tdb/legion/runtime/realm/transfer/channel.inl:166
#14 0x00001555364cd888 in Realm::BackgroundWorkManager::Worker::do_work (this=0x1554c81a59d0, max_time_in_ns=-1, interrupt_flag=0x0) at /lustre/scratch/vsyamaj/legion_s3d_tdb/legion/runtime/realm/bgwork.cc:599
#15 0x00001555364cb637 in Realm::BackgroundWorkThread::main_loop (this=0x125268b0) at /lustre/scratch/vsyamaj/legion_s3d_tdb/legion/runtime/realm/bgwork.cc:103
#16 0x00001555364cea88 in Realm::Thread::thread_entry_wrapper<Realm::BackgroundWorkThread, &Realm::BackgroundWorkThread::main_loop> (obj=0x125268b0) at /lustre/scratch/vsyamaj/legion_s3d_tdb/legion/runtime/realm/threads.inl:97
#17 0x0000155536684299 in Realm::KernelThread::pthread_entry (data=0x1301ec50) at /lustre/scratch/vsyamaj/legion_s3d_tdb/legion/runtime/realm/threads.cc:854
#18 0x0000155552afd2de in start_thread () from /lib64/libpthread.so.0
#19 0x000015555282ee83 in clone () from /lib64/libc.so.6
I made some changes recently and suspect Im doing something wrong somewhere, but not 100% sure that it isnt a bug in Realm either since the stack trace isnt super useful.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 30 (22 by maintainers)
Driver Version: 545.23.08I tried going back to
d11b7fccdbbut still cant reproduce it on sapling. I did make one change to work around #1725. Im going to try backing that out and see if it crashes on sapling.