legion: Legion: get_next_distributed_id and add_to_dependence_queue freeze
I’m seeing a freeze when running a cupynumeric application on Perlmutter. If I run a smaller problem size it seems to work, but running the full problem size I hit a freeze.
I have not been able to get a debug build of legate working on Perlmutter yet, but there are some stack traces here: https://sapling2.stanford.edu/~seshu/xpply5120/stacktraces/
It looks like there are threads stuck in 2 different places:
Thread 1 (Thread 0x7f0ddb8f8740 (LWP 1569219) "python"):
#0 0x00007f0ddbc5a70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f0cbabfd258 in ?? () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate
/core/_lib/data/../../../../../.././librealm-legate.so.1
#2 0x00007f0cbabfc050 in Realm::Event::external_wait_faultaware(bool&) const () from /pscratch/sd/s/seshu/cupynumeric-
nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#3 0x00007f0cbabfc53d in Realm::Event::external_wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/l
ib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#4 0x00007f0cbd4a084c in Legion::Internal::LgEvent::wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-0312
25/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#5 0x00007f0cbd61012e in Legion::Internal::ReplicateContext::get_next_distributed_id() () from /pscratch/sd/s/seshu/cu
pynumeric-nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#6 0x00007f0cbd7ab55a in Legion::Internal::ReplIndividualTask::create_future() () from /pscratch/sd/s/seshu/cupynumeri
c-nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#7 0x00007f0cbd8038f4 in Legion::Internal::IndividualTask::initialize_task(Legion::Internal::InnerContext*, Legion::Ta
skLauncher const&, Legion::Internal::Provenance*, bool, bool, std::vector<Legion::OutputRequirement, std::allocator<Leg
ion::OutputRequirement> >*) () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate/
core/_lib/data/../../../../../.././liblegion-legate.so.1
and
Thread 1 (Thread 0x7fa22fe6a740 (LWP 1788931) "python"):
#0 0x00007fa2301cc70c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fa10f1fd258 in ?? () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate
/core/_lib/data/../../../../../.././librealm-legate.so.1
#2 0x00007fa10f1fc050 in Realm::Event::external_wait_faultaware(bool&) const () from /pscratch/sd/s/seshu/cupynumeric-
nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#3 0x00007fa10f1fc53d in Realm::Event::external_wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/l
ib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././librealm-legate.so.1
#4 0x00007fa111aa084c in Legion::Internal::LgEvent::wait() const () from /pscratch/sd/s/seshu/cupynumeric-nightly-0312
25/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
#5 0x00007fa111bdb1ea in Legion::Internal::InnerContext::add_to_dependence_queue(Legion::Internal::Operation*, std::vector<Legion::StaticDependence, std::allocator<Legion::StaticDependence> > const*, bool, bool) () from /pscratch/sd/s/seshu/cupynumeric-nightly-031225/lib/python3.1/site-packages/legate/core/_lib/data/../../../../../.././liblegion-legate.so.1
Still trying to see if I can get a debug build of legate on Perlmutter, but its been very difficult.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 119 (108 by maintainers)
@PHHargrove they are in that same directory: https://sapling2.stanford.edu/~seshu/xpply5120/gasnet_traces/
@PHHargrove Sorry. Please try it now.
Example parsing outputs from a 2-rank run of a GASNet test:
Note the artificial
.txtsuffix was needed to get GitHub to allow me to attach.parse.pl.txt