legion: Realm segfault in startup on PowerPC

~This is another one that is for an older runtime, so it may not be actionable. I’ll keep this open until we can confirm that the bug has been fixed in a newer version.~

This replicates on master now. Legion version: c68a5d8cb5417283d5ed6adeccbc4c38d6c8ab03 (2025-01-08)

This is S3D on Lassen at 64 or 128 nodes. Crashes are non-deterministic, probability around 30%.

Here are some sample backtraces. Note these are from different runs.

[239] #9  <signal handler called>
[239] #10 __memcpy_power7 () at ../sysdeps/powerpc/powerpc64/power7/memcpy.S:356
[239] #11 0x0000200006845fa4 in gasnete_coll_p2p_memcpy_reqh_64 () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[239] #12 0x00002000067acc14 in gasnetc_processPacket () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[239] #13 0x00002000067af70c in gasnetc_poll_rcv_hca () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[239] #14 0x00002000067b0000 in gasnetc_poll_rcv_all () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[239] #15 0x000020000679a028 in gasnetc_AMPoll () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[239] #16 0x000020000678fdfc in _gasneti_AMPoll () at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/gasnet/release/include/gasnet_help.h:1528
[239] #17 _gasnet_AMPoll () at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/gasnet/release/include/gasnet_help.h:1528
[239] #18 Realm::GASNetEXWrapper::gex_am_poll () at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201
[239] #19 0x0000200005684a6c in Realm::GASNetEXPoller::do_work (this=0x1a43f530, work_until=...) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955
[239] #20 0x00002000054691ac in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x2000726aeb30, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/timers.inl:288
[239] #21 0x0000200005469cc0 in Realm::BackgroundWorkThread::main_loop (this=0x1e118320) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/bgwork.cc:103
[239] #22 0x00002000055de8d8 in Realm::KernelThread::pthread_entry (data=0x1e2f6b60) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/threads.cc:853
[239] #23 0x00002000007c8cd4 in start_thread (arg=0x2000726af8b0) at pthread_create.c:309
[239] #24 0x0000200000927f14 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104
[40] #9  <signal handler called>
[40] #10 __memcpy_power7 () at ../sysdeps/powerpc/powerpc64/power7/memcpy.S:356
[40] #11 0x0000200006845fa4 in gasnete_coll_p2p_memcpy_reqh_64 () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[40] #12 0x00002000067acc14 in gasnetc_processPacket () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[40] #13 0x00002000067af70c in gasnetc_poll_rcv_hca () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[40] #14 0x00002000067b0000 in gasnetc_poll_rcv_all () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[40] #15 0x000020000679a028 in gasnetc_AMPoll () from /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/language/build/lib/librealm.so.1
[40] #16 0x0000200006790508 in _gasneti_AMPoll () at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/gasnet/release/include/gasnet_help.h:1315
[40] #17 gasnete_wait_all (_numevents=16, _pevent=0x1f6d4120) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/gasnet/release/include/ibv-conduit/gasnet_extended.h:213
[40] #18 _gex_Event_WaitAll (_flags=0, _numevents=16, _pevent=0x1f6d4120) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/gasnet/release/include/ibv-conduit/gasnet_extended.h:221
[40] #19 Realm::GASNetEXWrapper::gex_coll_allgatherv (tm=0x1a739f00, val_in=0x1e660010, vals_out=0x1f6be810, bytes=<optimized out>, offsets=<optimized out>, flags=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:301
[40] #20 0x0000200005689188 in Realm::GASNetEXInternal::allgatherv (this=0x1a7199d0, val_in=0x1e660010 "\002", bytes=<optimized out>, vals_out=std::vector of length 88320, capacity 88320 = {...}, lengths=std::vector of length 256, capacity 256 = {...}) at /usr/tce/packages/gcc/gcc-12.2.1/rh/usr/include/c++/12/bits/stl_vector.h:1255
[40] #21 0x000020000566f13c in Realm::GASNetEXModule::allgatherv (this=<optimized out>, val_in=<optimized out>, bytes=<optimized out>, vals_out=..., lengths=...) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/gasnetex/gasnetex_module.cc:609
[40] #22 0x00002000055901c0 in Realm::allgather_announcement (dbs=..., targets=..., machine=0x1a719820, network_module=network_module@entry=0x1a719970) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/serialize.inl:179
[40] #23 0x00002000055a647c in Realm::RuntimeImpl::finish_configure (this=0x1a7184b0) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/realm/runtime_impl.cc:2414
[40] #24 0x0000200003728708 in Legion::Internal::Runtime::start (argc=<optimized out>, argv=<optimized out>, background=<optimized out>, supply_default_mapper=<optimized out>, filter=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/legion/runtime.cc:33669
[40] #25 0x00002000032a3228 in Legion::Runtime::start (argc=<optimized out>, argv=<optimized out>, background=<optimized out>, default_mapper=<optimized out>, filter=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/legion/runtime/legion/legion.cc:7236
[40] #26 0x00002000000c1690 in S3DRank::start_legion() () from /p/gpfs1/eslaught/s3d-subranks-2025-01-08-jan2025-legion/test_cases/0d_ammonia/try1_trial_run/ammonia_64_8/librhsf.so
[40] #27 0x00002000000bc2c8 in initialize_rhsf_legion_ () from /p/gpfs1/eslaught/s3d-subranks-2025-01-08-jan2025-legion/test_cases/0d_ammonia/try1_trial_run/ammonia_64_8/librhsf.so
[40] #28 0x00000000101f7728 in solve_driver (io=6) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/s3d/source/drivers/solve_driver.f90:194
[40] #29 0x00000000101f6ea4 in s3d () at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/s3d/source/drivers/main.f90:149
[40] #30 0x00000000100036f4 in main (argc=<optimized out>, argv=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-jan2025-legion/s3d/source/drivers/main.f90:8
[40] #31 0x0000200000825300 in generic_start_main (main=0x100036c0 <main>, argc=<optimized out>, argv=0x7fffffffa088, auxvec=0x7fffffffa980, init=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>, fini=<optimized out>) at ../csu/libc-start.c:266
[40] #32 0x00002000008254f4 in __libc_start_main (argc=<optimized out>, argv=<optimized out>, ev=<optimized out>, auxvec=<optimized out>, rtld_fini=<optimized out>, stinfo=<optimized out>, stack_on_entry=<optimized out>) at ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:81
[40] #33 0x0000000000000000 in ?? ()
[40] [Inferior 1 (process 92644) detached]
Crashes for an older (September 2024) version of Legion
[436] #9  <signal handler called>
[436] #10 __memcpy_power7 () at ../sysdeps/powerpc/powerpc64/power7/memcpy.S:387
[436] #11 0x0000200006734fa4 in gasnete_coll_p2p_memcpy_reqh_64 () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[436] #12 0x000020000669bc14 in gasnetc_processPacket () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[436] #13 0x000020000669e70c in gasnetc_poll_rcv_hca () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[436] #14 0x000020000669f000 in gasnetc_poll_rcv_all () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[436] #15 0x0000200006689028 in gasnetc_AMPoll () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[436] #16 0x000020000667edfc in _gasneti_AMPoll () at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/gasnet/release/include/gasnet_help.h:1528
[436] #17 _gasnet_AMPoll () at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/gasnet/release/include/gasnet_help.h:1528
[436] #18 Realm::GASNetEXWrapper::gex_am_poll () at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:201
[436] #19 0x000020000558d92c in Realm::GASNetEXPoller::do_work (this=0x1adcce60, work_until=...) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/gasnetex/gasnetex_internal.cc:2955
[436] #20 0x0000200005377024 in Realm::BackgroundWorkManager::Worker::do_work (this=this@entry=0x200065edeb30, max_time_in_ns=max_time_in_ns@entry=-1, interrupt_flag=interrupt_flag@entry=0x0) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/timers.inl:288
[436] #21 0x0000200005377b40 in Realm::BackgroundWorkThread::main_loop (this=0x21bdc860) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/bgwork.cc:103
[436] #22 0x00002000054e1740 in Realm::KernelThread::pthread_entry (data=0x21f8fcd0) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/threads.cc:854
[436] #23 0x00002000007c8cd4 in start_thread (arg=0x200065edf8b0) at pthread_create.c:309
[436] #24 0x0000200000927f14 in clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:104
[328] #9  <signal handler called>
[328] #10 __memcpy_power7 () at ../sysdeps/powerpc/powerpc64/power7/memcpy.S:387
[328] #11 0x0000200006734fa4 in gasnete_coll_p2p_memcpy_reqh_64 () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[328] #12 0x000020000669bc14 in gasnetc_processPacket () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[328] #13 0x000020000669e70c in gasnetc_poll_rcv_hca () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[328] #14 0x000020000669f000 in gasnetc_poll_rcv_all () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[328] #15 0x0000200006689028 in gasnetc_AMPoll () from /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/language/build/lib/librealm.so.1
[328] #16 0x000020000667f508 in _gasneti_AMPoll () at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/gasnet/release/include/gasnet_help.h:1315
[328] #17 gasnete_wait_all (_numevents=16, _pevent=0x230d3890) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/gasnet/release/include/ibv-conduit/gasnet_extended.h:213
[328] #18 _gex_Event_WaitAll (_flags=0, _numevents=16, _pevent=0x230d3890) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/gasnet/release/include/ibv-conduit/gasnet_extended.h:221
[328] #19 Realm::GASNetEXWrapper::gex_coll_allgatherv (tm=0x1b0cf200, val_in=0x230b8390, vals_out=0x23211dc0, bytes=<optimized out>, offsets=<optimized out>, flags=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/gasnetex/gasnetex_wrapper/gasnetex_wrapper.cc:301
[328] #20 0x00002000055920f8 in Realm::GASNetEXInternal::allgatherv (this=0x1afc4290, val_in=0x230b8390 "\001", bytes=<optimized out>, vals_out=std::vector of length 24516608, capacity 24516608 = {...}, lengths=std::vector of length 512, capacity 512 = {...}) at /usr/tce/packages/gcc/gcc-12.2.1/rh/usr/include/c++/12/bits/stl_vector.h:1255
[328] #21 0x0000200005577e1c in Realm::GASNetEXModule::allgatherv (this=<optimized out>, val_in=<optimized out>, bytes=<optimized out>, vals_out=..., lengths=...) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/gasnetex/gasnetex_module.cc:609
[328] #22 0x000020000548e418 in Realm::allgather_announcement (dbs=..., targets=..., machine=0x1afc40f0, network_module=network_module@entry=0x1afc4230) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/serialize.inl:179
[328] #23 0x00002000054a9680 in Realm::RuntimeImpl::finish_configure (this=0x1afc2d80) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/realm/runtime_impl.cc:2314
[328] #24 0x00002000036a5578 in Legion::Internal::Runtime::start (argc=<optimized out>, argv=<optimized out>, background=<optimized out>, supply_default_mapper=<optimized out>, filter=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/legion/runtime.cc:30407
[328] #25 0x000020000324c3e8 in Legion::Runtime::start (argc=<optimized out>, argv=<optimized out>, background=<optimized out>, default_mapper=<optimized out>, filter=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/legion/runtime/legion/legion.cc:7105
[328] #26 0x00002000000c1760 in S3DRank::start_legion() () from /p/gpfs1/eslaught/s3d-subranks-2025-01-08-sep2024-legion/test_cases/0d_ammonia/try1_trial_run/ammonia_128_3/librhsf.so
[328] #27 0x00002000000bc2c8 in initialize_rhsf_legion_ () from /p/gpfs1/eslaught/s3d-subranks-2025-01-08-sep2024-legion/test_cases/0d_ammonia/try1_trial_run/ammonia_128_3/librhsf.so
[328] #28 0x00000000101f7728 in solve_driver (io=6) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/s3d/source/drivers/solve_driver.f90:194
[328] #29 0x00000000101f6ea4 in s3d () at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/s3d/source/drivers/main.f90:149
[328] #30 0x00000000100036f4 in main (argc=<optimized out>, argv=<optimized out>) at /g/g19/eslaught/s3d-subranks-2025-01-08-sep2024-legion/s3d/source/drivers/main.f90:8
[328] #31 0x0000200000825300 in generic_start_main (main=0x100036c0 <main>, argc=<optimized out>, argv=0x7fffffff9ca8, auxvec=0x7fffffffa5a0, init=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>, fini=<optimized out>) at ../csu/libc-start.c:266
[328] #32 0x00002000008254f4 in __libc_start_main (argc=<optimized out>, argv=<optimized out>, ev=<optimized out>, auxvec=<optimized out>, rtld_fini=<optimized out>, stinfo=<optimized out>, stack_on_entry=<optimized out>) at ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:81
[328] #33 0x0000000000000000 in ?? ()

Legion version is a branch based on a September 2024 commit, plus one fix: https://gitlab.com/StanfordLegion/legion/-/commit/d4392d6f01762cdde198a8f44da8c05643c9ea34

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 49 (49 by maintainers)

Most upvoted comments

Applying the patch at https://github.com/StanfordLegion/legion/issues/1821#issuecomment-2605908483 appears to have fixed the crash.

I created a PR at https://github.com/StanfordLegion/gasnet/pull/38 to track this patch since it seems to be important for RISC-like systems. I’ll wait for some more confirmation before merging, but at least it’s available on a short-term basis for anyone else who finds a need to use it.

The admins confirmed the I/O issue on the machine and said a fix was put in place. I have resubmitted the jobs and am now waiting for them to run again.

I’ve had runs queued all week, but the system seems be failing in job launch, so I don’t have any meaningful results at the moment. I hope that the admins will resolve the issue so that I can run jobs going forward.

Yup, I’m just waiting for jobs to go through at this point. Everything is queued.

The only update from yesterday is that I now have release mode jobs with the patch that have passed up to 16 nodes. Remaining jobs are in the queue.

Runs with the patch in debug GASNet have now completed successfully out to 64 nodes. Previously (out of a set of 10 runs) I had 2 failures at this node count.

Edit: to clarify:

  • Previously in debug mode I was hitting errors about 90% of the time at 16 nodes. That failure mode is clearly gone now. The debug checks are happy.
  • Previously in release mode I was hitting segfaults about 20% of the time at 64 nodes. This failure mode may or may not be gone, because debug mode may impact the timing of the crash.

So far in a debug build of GASNet, the patch in https://github.com/StanfordLegion/legion/issues/1821#issuecomment-2589153502 is passing up to 16 nodes. I’ll return in the morning to see how we do as we scale a bit farther, though I’m not sure I want to take debug GASNet all the way out.

I am investigating right now, but not optimistic about having a solution tonight.

In the meantime, the 1-line removal in the patch below probably addresses the root cause of the error message from the debug build. However, I will need additional time (not tonight) to convince myself this change cannot lead to deadlock and to look for lower-impact solution(s).

@elliottslaughter I would appreciate it if you can recompile your debug build with this applied and report back either resolution of the original problem (my hope 🤞) or the new crash stack.

--- a/ibv-conduit/gasnet_core_sndrcv.c
+++ b/ibv-conduit/gasnet_core_sndrcv.c
@@ -1145,7 +1145,6 @@ gasnetc_cep_t *gasnetc_bind_cep_inner(gasnetc_EP_t ep, gasnetc_epid_t epid, gasn
       if (should_poll_rcv && !gasnetc_sema_read(GASNETC_CEP_SQ_SEMA(_cep))) {  \
         gasnetc_poll_rcv_all(_ep, GASNETC_RCV_REAP_LIMIT GASNETI_THREAD_PASS); \
         MAYBE_POLL_RCV_PSHM();                                                 \
-        GASNETI_PROGRESSFNS_RUN();                                             \
       }                                                                        \
     } while (0)
 #else