legion: Reductions cause a deadlock when using Legion with CUDA

One of our codes using FleCSI with the Legion backend consistently runs into a deadlock when performing a reduction on a field. The code gets stuck when trying to get the result of the reduction without actually launching the reduction task itself. It only occurs when at least two ranks are involved, a ghost copy had to occur some time before, and the tasks are launched on a GPU with CUDA. This happens on master and the current 24.03.0 release.

Unfortunately I can’t share a full reproducer but we could internally narrow it down to merge commit dde5f0c30cbaaf67fa47b32364f5976c6f1d1aa7. We were using the control replication branch before that was merged.

In particular we found that changing supports_scatter_gather_path in runtime/realm/cuda/cuda_internal.cc to always return false makes the problem consistently go away.

I’m happy to try out any fixes you come up with.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 30 (17 by maintainers)

Most upvoted comments

@apryakhin I’ve run the apriakhin/gather-align-ib branch of Legion and everything still works fine. Thank you!

@tukss Can you please do another run on the following branch?

  • gather-align-ib

It’s a better patch which passes on the tests I wrote that suppose to replicate the type of operations you are running. However, since I can’t run your exact tests directly just want to make sure it fixes everything for you.

@tukss Thanks for verifying it. The patch will have to go through the peer view (hopefully this week) and I will make sure we get it merged for 24.06.

Sure. Here are the logs @apryakhin.

crashlog.txt

There is a bug in the size alignment of the staging buffer. Essentially gpu gather moves first the indirection through the IB and then stages results of the actual gather through another IB. The size of the second IB isn’t aligned with the field_size=24. That leads to the under counting of the actual data points and therefore the gpu gather never completes.

The fix is in the local branch already. I am going to sit on it for some short time to make sure that’s the right fix and that there is the right test for it.