triton: [Issue]: Triton Compiler Takes Indefinite Time in ttgir -> llir Stage.

Problem Description

Full source code to reproduce: rep.py.gz

Triton version: upstream d688063f731cfc4d9431bb8c0d0d73dce8cd1c38 Docker Container: rocm/pytorch-private:compute-rocm-rel-6.1-116_ubuntu22.04_py3.9_pytorch_rocm6.1_internal_testing_ae01701

Can be reproduced in both MI200(gfx90a) and Navi3x. Debugging print shows the compiler hangs during ttgir->llir stage.

Operating System

Ubuntu 22.04.4 LTS (Jammy Jellyfish)

CPU

AMD Ryzen Threadripper PRO 5975WX 32-Cores

GPU

AMD Instinct MI210

ROCm Version

ROCm 6.1.0

ROCm Component

No response

Steps to Reproduce

Download the rep.py.gz in the Description section, and then

gunzip rep.py.gz
python rep.py

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (1 by maintainers)

Commits related to this issue

Most upvoted comments

I am working on this, because it looks (very) slightly simpler than https://github.com/ROCm/triton-internal/issues/104

This is what I got so far:

  • The bug (as mentioned previously) originates from convert-builtin-func-to-llvm
  • The main issue for this specific case, is about the stores. I made a reproduction bug.mlir which is the file just before convert-builtin-func-to-llvm (attached to this comment) and I commented out all the loads and some of the stores. While triton-opt terminates, the output produced is massively large. The more stores we add back into bug.mlir, the more time it takes to complete (I think that if we leave it long enough it will eventually complete)
  • The source of the issue looks like the mergeIdenticalBlocks transformation contained in the simplifyRegion utility. If I disable that transformation enableRegionSimplify=false then compilation is quite quick.
  • I produced a disable_simplify.mlir output that comes from bug.mlir when passing enableRegionSimplify=false to the rewriter. If we do: triton-opt --canonicalize disable_simplify.mlir we see the same masssive output as befreo with triton-opt that takes some time to finish. Instead, if we do: triton-opt --canonicalize="region-simplify=false" disable_simplify.mlir the output is normal, and triton-opt terminates quickly.

repro.zip

Does the rewriter call simplifyRegions (and probably other passes to canonicalize stuff) after it matches and rewrites all the ops?

Yes

We only need to set it for the rewriter in builtin_func_to_llvm pass. right? If so, are there any other drawbacks ?

And anytime we invoke the rewriter after that. I see that builtin_func_to_llvm is the last pass, so it shouldn’t be an issue.

Of course there is the core drawback that we will disable block merging in all cases. But this is something we can worry later (and I will try to work on it in my “spare” time)