legion: S3D freeze at large node counts

As suspected by Mike in our meeting today, we are seeing freezes in S3D at large node counts. This is a preliminary report because I just encountered the issue and am still getting data.

What I know so far:

  • My most recent job on the 24.03.0 rc branch froze and 2048 and 8192 nodes, but ran at 4096 nodes
  • A previous job with my hacked up branch mentioned in https://github.com/StanfordLegion/legion/issues/1653#issuecomment-2005732308 froze at 8192 nodes but worked at all other node counts. Edit: I have now seen freezes at 2048 and 4096 nodes as well.
  • To the best of my knowledge, we have never been able to run a job past 8192 nodes. When Seshu tried (I can’t remember if this was February 2024 or November 2023), he hit symptoms that looked like a freeze. It’s possible these are all the same symptoms, but we didn’t investigate at the time so it’s hard to be sure

I have backtraces from the 2048 node run I just did. Don’t blame me for not having -ll:force_kthreads; I was not expecting the freeze and wasn’t prepared to collect backtraces, so they are what they are. From an initial scan I haven’t see anything interesting, so likely I will need to rerun with -ll:force_kthreads.

http://sapling.stanford.edu/~eslaught/s3d_freeze_debug2/bt2048/

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 41 (30 by maintainers)

Most upvoted comments

@PHHargrove Thanks for the explanation! I don’t want to trouble you with too many questions. I will just poke around one last possibility.

The progress of a sending process also depends on its target process having receive buffers posted/completion queue polled, so only the sending process doing “comprehensive” progress is not enough. I see gasnetex is using two sets of endpoints and completion queues (one for request and one for reply). If all threads in process A sending requests to process B (so it only polls the request ep/cq) but process B is sending replies to process A (so it only polls the reply ep/cq), could this lead to a deadlock?

Just found it also polls the reply ep/cq when it is sending requests. Nevermind.

@JiakunYan Together, @bonachea and I have looked previously at the libfabric documentation and we determined that it allows for the (unfortunate) behavior you describe. However, we do not know the behavior of any particular providers (such as cxi) nor would there be any guarantee that the behavior would not change in a future release of libfabric.

Checked with the UCX team and we should be fine with UCX. ucp_worker_progress will not block the calling thread. Of course, this is not applicable/related to Slingshot in any way as UCX does not have support for Slingshot.