triton: [Issue]: Triton Compiler Takes Indefinite Time in ttgir -> llir Stage.
Problem Description
Full source code to reproduce: rep.py.gz
Triton version: upstream d688063f731cfc4d9431bb8c0d0d73dce8cd1c38
Docker Container: rocm/pytorch-private:compute-rocm-rel-6.1-116_ubuntu22.04_py3.9_pytorch_rocm6.1_internal_testing_ae01701
Can be reproduced in both MI200(gfx90a) and Navi3x. Debugging print shows the compiler hangs during ttgir->llir stage.
Operating System
Ubuntu 22.04.4 LTS (Jammy Jellyfish)
CPU
AMD Ryzen Threadripper PRO 5975WX 32-Cores
GPU
AMD Instinct MI210
ROCm Version
ROCm 6.1.0
ROCm Component
No response
Steps to Reproduce
Download the rep.py.gz in the Description section, and then
gunzip rep.py.gz
python rep.py
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (1 by maintainers)
Commits related to this issue
- Mitigate compiler bug (https://github.com/ROCm/triton/issues/596) — committed to ROCm/aotriton by xinyazhang 2 years ago
- Mitigate compiler bug (https://github.com/ROCm/triton/issues/596) — committed to ROCm/aotriton by xinyazhang 2 years ago
I am working on this, because it looks (very) slightly simpler than https://github.com/ROCm/triton-internal/issues/104
This is what I got so far:
convert-builtin-func-to-llvmbug.mlirwhich is the file just beforeconvert-builtin-func-to-llvm(attached to this comment) and I commented out all the loads and some of the stores. Whiletriton-optterminates, the output produced is massively large. The more stores we add back intobug.mlir, the more time it takes to complete (I think that if we leave it long enough it will eventually complete)mergeIdenticalBlockstransformation contained in thesimplifyRegionutility. If I disable that transformationenableRegionSimplify=falsethen compilation is quite quick.disable_simplify.mliroutput that comes frombug.mlirwhen passingenableRegionSimplify=falseto the rewriter. If we do:triton-opt --canonicalize disable_simplify.mlirwe see the same masssive output as befreo withtriton-optthat takes some time to finish. Instead, if we do:triton-opt --canonicalize="region-simplify=false" disable_simplify.mlirthe output is normal, andtriton-optterminates quickly.repro.zip
Update 06/05/2024
We have those two passes in upstream
The hanging happens in the add_builtin_func_to_llvmir pass: https://github.com/triton-lang/triton/blob/fa2271e37f4e0ccfa8829501e40533060937cfe5/third_party/amd/backend/compiler.py#L185
Yes
And anytime we invoke the rewriter after that. I see that builtin_func_to_llvm is the last pass, so it shouldn’t be an issue.
Of course there is the core drawback that we will disable block merging in all cases. But this is something we can worry later (and I will try to work on it in my “spare” time)