legion: Legion: get_next_distributed_id and add_to_dependence_queue freeze

I’m seeing a freeze when running a cupynumeric application on Perlmutter. If I run a smaller problem size it seems to work, but running the full problem size I hit a freeze.

I have not been able to get a debug build of legate working on Perlmutter yet, but there are some stack traces here: https://sapling2.stanford.edu/~seshu/xpply5120/stacktraces/

It looks like there are threads stuck in 2 different places:

Thread 1 (Thread 0x7f0ddb8f8740 (LWP 1569219) "python"):
#0  0x00007f0ddbc5a70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0cbabfd258 in ?? () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate
/core/_lib/data/../../../../../.././librealm-legate.so.1
#2  0x00007f0cbabfc050 in Realm::Event::external_wait_faultaware(bool&) const () from /pscratch/sd/s/seshu/cupynumeric-
nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#3  0x00007f0cbabfc53d in Realm::Event::external_wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/l
ib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#4  0x00007f0cbd4a084c in Legion::Internal::LgEvent::wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-0312
25/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#5  0x00007f0cbd61012e in Legion::Internal::ReplicateContext::get_next_distributed_id() () from /pscratch/sd/s/seshu/cu
pynumeric-nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#6  0x00007f0cbd7ab55a in Legion::Internal::ReplIndividualTask::create_future() () from /pscratch/sd/s/seshu/cupynumeri
c-nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#7  0x00007f0cbd8038f4 in Legion::Internal::IndividualTask::initialize_task(Legion::Internal::InnerContext*, Legion::Ta
skLauncher const&, Legion::Internal::Provenance*, bool, bool, std::vector<Legion::OutputRequirement, std::allocator<Leg
ion::OutputRequirement> >*) () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate/
core/_lib/data/../../../../../.././liblegion-legate.so.1

and

Thread 1 (Thread 0x7fa22fe6a740 (LWP 1788931) "python"):
#0  0x00007fa2301cc70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fa10f1fd258 in ?? () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate
/core/_lib/data/../../../../../.././librealm-legate.so.1
#2  0x00007fa10f1fc050 in Realm::Event::external_wait_faultaware(bool&) const () from /pscratch/sd/s/seshu/cupynumeric-
nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#3  0x00007fa10f1fc53d in Realm::Event::external_wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/l
ib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#4  0x00007fa111aa084c in Legion::Internal::LgEvent::wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-0312
25/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#5  0x00007fa111bdb1ea in Legion::Internal::InnerContext::add_to_dependence_queue(Legion::Internal::Operation*, std::vector<Legion::StaticDependence, std::allocator<Legion::StaticDependence> > const*, bool, bool) () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1

Still trying to see if I can get a debug build of legate on Perlmutter, but its been very difficult.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 119 (108 by maintainers)

Most upvoted comments

@PHHargrove Sorry. Please try it now.

Example parsing outputs from a 2-rank run of a GASNet test:

$ ./parse.pl trace_*
# SRC	DST	SEND	RECV
0	1	33606	33606
1	0	33605	33605

Note the artificial .txt suffix was needed to get GitHub to allow me to attach.
parse.pl.txt