TheRock: [Windows]: HIPRTC_ERROR_COMPILATION on gfx1201
Problem Description
Not sure if I missed something or we are still is lacking gfx1201 support.
At least test_conv_transpose2d test says Unsupported PAL device with ASIC revision #62 which is something. But the other test_conv_cudnn_nhwc_support, just gives generic compilation error without much more info. (See results in additional info section)
I did default build with cmake -GNinja .. -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTHEROCK_AMDGPU_FAMILIES=gfx1201 -DTHEROCK_AMDGPU_WINDOWS_INTEROP_DIR="E:\dev\amdgpu-windows-interop"
Also interesting is that when MIOpenDriver.exe is run as part of the test, it gives different result. I don’t know how it works, maybe you pass things in environment variable, but
Running MIOpenDriver.exe conv -n 1 -c 8 -H 5 -W 5 -k 4 -y 3 -x 3 -p 1 -q 1 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 2 -t 1 standalone crashes after it fails to find device. I would suggest just aborting/exiting earlier if such condition happens, instead of crashing later.
[0x0] MIOpen!miopen::Handle::GetDeviceName+0xf 0x59f17fe890 0x7ff875940017
[0x1] MIOpen!miopen::solver::conv::GemmFwdBase::IsApplicable+0xa7 0x59f17fe8e0 0x7ff87594363f
[0x2] MIOpen!miopen::solver::conv::GemmFwd1x1_0_1::IsApplicable+0xf 0x59f17fead0 0x7ff875b9128b
[0x3] MIOpen!miopen::solver::SolverContainer<miopen::solver::conv::GemmFwd1x1_0_1,miopen::solver::conv::GemmFwd1x1_0_1_int8,miopen::solver::conv::GemmFwd1x1_0_2,miopen::solver::conv::GemmFwdRest,miopen::solver::conv::GemmBwd1x1_stride1,miopen::solver::conv::GemmBwd1x1_stride2,miopen::solver::conv::GemmBwdRest,miopen::solver::conv::GemmWrw1x1_stride1,miopen::solver::conv::GemmWrwUniversal>::GetWorkspaceSizes<miopen::ExecutionContext,miopen::conv::ProblemDescription>::<lambda_1>::operator()<miopen::solver::conv::GemmFwd1x1_0_1>+0xdb 0x59f17feb10 0x7ff875b910de
[0x4] MIOpen!miopen::each_args<`lambda at E:\dev\TheRock\ml-libs\MIOpen\src\include\miopen\find_solution.hpp:403:13',miopen::solver::conv::GemmFwd1x1_0_1,miopen::solver::conv::GemmFwd1x1_0_1_int8,miopen::solver::conv::GemmFwd1x1_0_2,miopen::solver::conv::GemmFwdRest,miopen::solver::conv::GemmBwd1x1_stride1,miopen::solver::conv::GemmBwd1x1_stride2,miopen::solver::conv::GemmBwdRest,miopen::solver::conv::GemmWrw1x1_stride1,miopen::solver::conv::GemmWrwUniversal>+0x1e 0x59f17fec90 0x7ff875b84c63
[0x5] MIOpen!miopen::solver::SolverContainer<miopen::solver::conv::GemmFwd1x1_0_1,miopen::solver::conv::GemmFwd1x1_0_1_int8,miopen::solver::conv::GemmFwd1x1_0_2,miopen::solver::conv::GemmFwdRest,miopen::solver::conv::GemmBwd1x1_stride1,miopen::solver::conv::GemmBwd1x1_stride2,miopen::solver::conv::GemmBwdRest,miopen::solver::conv::GemmWrw1x1_stride1,miopen::solver::conv::GemmWrwUniversal>::GetWorkspaceSizes<miopen::ExecutionContext,miopen::conv::ProblemDescription>+0x113 0x59f17fed10 0x7ff875b7fa70
[0x6] MIOpen!AllGemmWorkspaceSize+0x20 0x59f17fee30 0x7ff874d0c864
[0x7] MIOpen!miopen::`anonymous namespace'::GetWorkSpaceSizeGEMM+0x94 0x59f17fee70 0x7ff874d0b969
[0x8] MIOpen!miopen::ConvolutionDescriptor::GetWorkSpaceSize+0x959 0x59f17feed0 0x7ff874d23a4f
[0x9] MIOpen!miopenConvolutionBackwardDataGetWorkSpaceSize::<lambda_1>::operator()+0x2cf (Inline Function) (Inline Function)
[0xa] MIOpen!miopen::try_+0x2cf (Inline Function) (Inline Function)
[0xb] MIOpen!miopenConvolutionBackwardDataGetWorkSpaceSize+0x2ff 0x59f17ff090 0x7ff6b254b3df
[0xc] MIOpenDriver!ConvDriver<float,float>::AllocateBuffersAndCopy+0xb0f 0x59f17ff7b0 0x7ff6b29cde26
[0xd] MIOpenDriver!main+0x206 0x59f17ffb90 0x7ff6b346a9c4
[0xe] MIOpenDriver!invoke_main+0x22 (Inline Function) (Inline Function)
[0xf] MIOpenDriver!__scrt_common_main_seh+0x10c 0x59f17ffc80 0x7ff959b5e8d7
[0x10] KERNEL32!BaseThreadInitThunk+0x17 0x59f17ffcc0 0x7ff95abdc5dc
[0x11] ntdll!RtlUserThreadStart+0x2c 0x59f17ffcf0 0x0
Log before the above crash
:1:paldevice.cpp :915 : 176315778463 us: [pid:20520 tid: 0x54632] Unsupported PAL device with ASIC revision #62
:3:comgrctx.cpp :33 : 176315778692 us: [pid:20520 tid: 0x54632] Loading COMGR library.
:3:comgrctx.cpp :128 : 176315778747 us: [pid:20520 tid: 0x54632] Loaded COMGR library version 3.0.
:3:hip_context.cpp :54 : 176315778860 us: [pid:20520 tid: 0x54632] Direct Dispatch: 0
:3:hip_stream.cpp :281 : 176315778887 us: [pid:20520 tid: 0x54632] hipStreamCreate ( 00000059F17FFA58 )
:3:hip_stream.cpp :281 : 176315778909 us: [pid:20520 tid: 0x54632] hipStreamCreate: Returned hipErrorNoDevice :
:3:hip_context.cpp :166 : 176315778941 us: [pid:20520 tid: 0x54632] hipInit ( 0 )
:3:hip_context.cpp :166 : 176315778959 us: [pid:20520 tid: 0x54632] hipInit: Returned hipErrorNoDevice :
:3:hip_device_runtime.cpp :671 : 176315778980 us: [pid:20520 tid: 0x54632] hipGetDevice ( 00000059F17FF920 )
:3:hip_device_runtime.cpp :671 : 176315778995 us: [pid:20520 tid: 0x54632] hipGetDevice: Returned hipErrorNoDevice :
MIOpen Error: E:/dev/TheRock/ml-libs/MIOpen/src/hip/handlehip.cpp:150: No device
MIOpen(HIP): Info [] MIOPEN_FIND_MODE = NORMAL(1)
MIOpen(HIP): Info [AmdRocmMetadataVersionDetect] ROCm MD version AMDHSA_COv3, HIP version 6.5.25204, MIOpen version 3.4.0.204b36923
MIOpen(HIP): Info2 [GetWorkSpaceSize]
MIOpen(HIP): Info2 [GetWorkspaceSizes] fft: Not applicable
<crash>
Operating System
10.0.26100
CPU
AMD Ryzen 9 7950X 16-Core Processor
GPU
AMD Radeon RX 9070 XT
ROCm Version
TheRock a742718b
ROCm Component
HIP
Steps to Reproduce
Run python -m pytest -v smoke-tests on gfx1201
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
Test results
smoke-tests/pytorch_smoke_test.py::TestROCmAvailability::test_rocm_available PASSED [ 9%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_multiplication PASSED [ 18%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_batch_matrix_multiplication PASSED [ 27%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_multiplication_at_operator PASSED [ 36%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_elementwise_multiplication PASSED [ 45%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_transpose PASSED [ 54%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_dot_product PASSED [ 63%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_vector_multiplication PASSED [ 72%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_multiplication_matmul PASSED [ 81%]
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_transpose2d FAILED [ 90%]
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_cudnn_nhwc_support FAILED [100%]
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_transpose2d
Interesting part of log:
MIOpen(HIP): Info2 [LoadBinary] Unable to load binary for: "naive_conv.cpp.obj"; args: -DMIOPEN_USE_FP16=0 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=1 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201
MIOpen(HIP): Info [PrintVersion] HIPRTC v.9.0
MIOpen(HIP): Trace [BuildHip] HIPRTC compile ROCm path: E:\dev\TheRock\build\dist\rocm\
:1:paldevice.cpp :915 : 175388348189 us: [pid:5720 tid: 0x56464] Unsupported PAL device with ASIC revision #62
:3:comgrctx.cpp :33 : 175388348340 us: [pid:5720 tid: 0x56464] Loading COMGR library.
:3:comgrctx.cpp :128 : 175388348376 us: [pid:5720 tid: 0x56464] Loaded COMGR library version 3.0.
:3:hip_context.cpp :54 : 175388348456 us: [pid:5720 tid: 0x56464] Direct Dispatch: 0
:3:hip_device_runtime.cpp :671 : 175388348470 us: [pid:5720 tid: 0x56464] hipGetDevice ( 000000CEBDFE70D4 )
:3:hip_device_runtime.cpp :671 : 175388348476 us: [pid:5720 tid: 0x56464] hipGetDevice: Returned hipErrorNoDevice :
MIOpen(HIP): Error [Compile] 'hiprtcCompileProgram(prog.get(), c_options.size(), c_options.data())' naive_conv.cpp: HIPRTC_ERROR_COMPILATION (6)
MIOpen(HIP): Error [BuildHip] HIPRTC status = HIPRTC_ERROR_COMPILATION (6), source file: naive_conv.cpp
MIOpen(HIP): Warning [BuildHip] warning: HIPRTC error log empty
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_cudnn_nhwc_support
Interesting part of log, note the difference from previous one, there are no logs from HIP at all:
MIOpen(HIP): Trace [BuildHip] HIPRTC compile ROCm path: E:\dev\TheRock\build\dist\rocm\
MIOpen(HIP): Error [Compile] 'hiprtcCompileProgram(prog.get(), c_options.size(), c_options.data())' naive_conv.cpp: HIPRTC_ERROR_COMPILATION (6)
MIOpen(HIP): Error [BuildHip] HIPRTC status = HIPRTC_ERROR_COMPILATION (6), source file: naive_conv.cpp
MIOpen(HIP): Warning [BuildHip] warning: HIPRTC error log empty
MIOpen(HIP): Info2 [StoreRecord] Trying to store record at key 16-1-1-3x3-8-1-1-1-1x1-2x1-1x1-0-NCHW-FP32-F in cache for file "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [StoreRecordUnsafe] Storing record: 16-1-1-3x3-8-1-1-1-1x1-2x1-1x1-0-NCHW-FP32-F
MIOpen(HIP): Info2 [FindRecordUnsafe] Looking for key 16-1-1-3x3-8-1-1-1-1x1-2x1-1x1-0-NCHW-FP32-F in file "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [UpdateDbModificationTime] Updating db modification time for "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [ValidateUnsafe] DB file is newer than cache: 175388367555100, 175388364048400
MIOpen(HIP): Info2 [UpdateDbModificationTime] Updating db modification time for "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [Measure] Db::StoreRecord time: 0.6867 ms
MIOpen Error: E:/dev/TheRock/ml-libs/MIOpen/src/hipoc/hipoc_program.cpp:299: Code object build failed. Source: naive_conv.cpp
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 77 (40 by maintainers)
I did some hipblaslt tuning, and it looks like it’s absolutely necessary if you want to get reasonable performance. Some rough comparison.
@jammm at some point, I would like a standalone script that runs some performance diagnostics and runs some key benchmarks. We’ve got this embedded in our qa procedures, but every time we go to support this is the field, we have a bunch of back and forths like this one. I’d love to just have first level performance debugging to have step 1 be “run this script and paste results”.
If we had that, I daresay it would short circuit half of the field support issues I see on numerous forums.
Maybe:
validate_amd_pytorch_goes_vroom.pyHeads up that if MIOpen is reporting that it’s using the Winograd solver, from the above:
miopenSp3AsmConvFury_v2_4_1_gfx11_1536vgprs_fp16_fp16acc_f2x3_c32_stride1
This means MIOpen isn’t using explicit GEMM for this. If not using explicit GEMM then there is no use of Im2col or a GEMM from rocBLAS or hipBLASLt so no tuning of those will help in this case. Though in general MIOpen is going to try and pick between multiple solver choices which on this hardware likely includes:
MIOpen also has a number of implicit GEMM based solvers but not many of these are available in Windows yet but we’re working on it.
Happy to land it in hipblaslt as well.
For me it’s
With
MIOPEN_GEMM_ENFORCE_BACKEND=1:Im2d2Col_v2intoCijk_Ailk_Bljk_HHS_BH_MT64x32x8_SN_1LDSB0_AMAS2_BL0_BS1_EPS0_GLVWA2_GLVWB1_GRVW2_GSU1_GSUASB_ISA000_IU1_K1_KLS_LBSPPA0_LBSPPB0_LPA0_LPB0_LRVW2_MIAV0_MMFSC_NLCA1_NLCB1_PGR0_PLR1_SIA1_SS0_SU32_SUM0_SUS256_SVW4_TT4_2_TLDS0_UMLDSA0_UMLDSB0_USFGRO0_VAW2_VS1_VW2_VWB2_WSGRA0_WSGRB0_WS64_WG16_16_1_WGM8With
MIOPEN_GEMM_ENFORCE_BACKEND=5:Im2d2Col_v2intoCijk_Ailk_Bljk_HHS_BH_UserArgs_MT16x16x64_MI16x16x1_SN_LDSB0_AFC0_AFEM1_AFEM1_ASEM1_CLR1_CADS0_DTVA0_DTVB0_EPS0_FDSI0_GRPM1_GRVWA8_GRVWB8_GSUAMB_GLS0_ISA1201_IU1_K1_LBSPPA256_LBSPPB256_LBSPPM0_LPA16_LPB16_LPM0_LRVW8_LWPMn1_MIAV1_MIWT1_1_MO40_NTn1_NTA0_NTB0_NTC0_NTD0_NTM0_NEPBS0_NLCA1_NLCB1_ONLL1_PGR0_PLR0_PKA0_SIA3_SS0_SPO0_SRVW0_SSO0_SVW8_SK0_SKXCCM0_TLDS0_ULSGRO0_USL1_UIOFGRO0_USFGROn1_VSn1_VWA1_VWB1_WSGRA0_WSGRB0_WS32_WG16_2_1and the difference difference in perf is significant. Notice that it has
UserArgs, because hipblaslt has some gfx1201 tuning files in it.Yeah, on gfx1151, it uses https://github.com/ROCm/MIOpen/blob/66aec60520122a1972cb4914bc7a9c59ceede69d/src/solver/conv/conv_wino_fury_RxS.cpp for this convolution.
And looking at this https://github.com/ROCm/MIOpen/blob/66aec60520122a1972cb4914bc7a9c59ceede69d/src/solver/conv/conv_wino_fury_RxS.cpp#L254-L256 it’s seems to be only for gfx11… which well, maybe is true, or maybe is just not updated for newer arch. (there are winograd solvers for gfx9, gfx10, gfx11, not yet updated for gfx12 it seems)
Honestly those hardcoded and scattered around checks in various ROCm parts are getting quite tiresome to track.
I guess I should buy gfx950, which is blessed and tested target.
Maybe it will help with overall performance. But as I understand it should currently use the same tuning in both Native Windows and WSL2 case. So I suspect there is still some other config difference between those two.
That’s our problem:
And when I set
TORCH_BLAS_PREFER_CUBLASLT=1I get warning that it’s not supported.EDIT:
Forgot to say, on WSL it’s using
_BlasBackend.CublasltThat could be solved by
pip install "numpy<2"but it shouldn’t be the cause of the TDR’s.I’m not sure what’s causing your issue. Perhaps @kasper93 is right and the two ROCm’s are clashing against each other somehow. Can you try temporarily renaming the ROCm folder you installed yourself and try again perhaps?
Thank you for insight. I’m focused only on gfx12 currently and none of Winograd solvers are supported yet. I’m messing around with GEMM and differences between rocBLAS or hipBLASLt. But indeed this Winograd kernel was a reason why @jammm didn’t see performance difference in workloads where I’ve seen it, because it was always using kernel from miopen directly.
@jammm @scottt : The performance difference I’m seeing is because rocBLAS is built without hipBLASLt. See BUILD_WITH_HIPBLASLT in rocBLAS.
That’s also why
ROCBLAS_USE_HIPBLASLTdoesn’t work to force it, and either way it would use hipBLASLt by default for this case. MIOpen always calls rocBLAS (it’s default backend for GEMM).If I set MIOPEN_GEMM_ENFORCE_BACKEND=5 to force MIOpen to call hipBLASLt directly, I get the same performance on both builds in all test cases.
MIOPEN_GEMM_ENFORCE_BACKENDunsetMIOPEN_GEMM_ENFORCE_BACKEND=5Hey @jammm im from the shark discord.
I think your post on X about the 60it/s is not quite right at least when I test it. It seems that comfy reports the wrong it/s for euler because the gpu still keeps going after then gen is already at 50/50 in the console. If you look at the UI with euler it takes a long time for the image to appear. When testing other samplers that are also as fast as euler like dpmpp_2m it takes no time for the image to appear and the overall time is the same. Meaning that euler reports as finished but in reality its not thats why the image takes so long to appear because its still being sampled.
I hear the audible coil whine still going after the image is already finished in the terminal.
Interesting. I also tried
--gpu-onlyand now on the gfx1151 it goes to 70-80 it/sHmm, there seem to be some GPU/CPU synchronization “fun”. On WSL2 it actually doesn’t change much, but on native windows, using
--gpu-onlywhich keeps everything in VRAM and run text encoders on GPU too. Makes significant difference on new pytorch 2.7. Not sure if this is “Windows” thing or some other factor going from 2.6 to 2.7 pytorch, because on WSL2 it’s 2.6.FYI, I sent some fixes for ComfyUI, https://github.com/comfyanonymous/ComfyUI/pull/8289 . On WSL2 I get VAE speed up and this dreaded OOM on VAE Decode is no more. Though TDR is still there on huge sizes.
Note that this is not related to perf difference between builds discussed in this thread, it’s just making ComfyUI faster in general.
@jammm , besides investigating why our build of MIOpen does not use hipBLASLt, note that our first cut of validate_torch_vroom.py did not capture the perf difference that @kasper93 is seeing with
v1-5-pruned-emaonly.safetensors512x512.One idea to improve the latter, is we do some Pytorch level profiling of end-to-end models runs like sdxl-base-1.0.py , capture the input parameters and shapes of the critical stages and measure those operations in
validate-torch-perf.py.See the logs from the MIOpen below, there isn’t really much difference. The main thing, is that HIPBLASLT logs are missing from slow (Windows) build,
[HIPBLASLT][72350][Api][rocblaslt_algo_get_heuristic_cpp] returnAlgoCount=1.Notable difference is that at the beginning of the execution there are two
rocblaslt_createcalls in “fast” build and single call in "slow’ build, which may suggest that some component is not creating own handle in Windows case.What I did initially is to benchmark some common primitives and this conv was one of the things that stood up when comparing both builds. There were other things, but one at the time…
I wanted to get
hipblaslt-benchcommand, but on Windows buildHIPBLASLT_LOG_MASK=32doesn’t work for this test case, while it does on WSL2 build. AlsoROCBLAS_USE_HIPBLASLTdoesn’t seem to affect this case.(logs stripped from clarity)
Windows:
WSL2:
I don’t see any meaningful difference in those tests.
But if I change
conv2dto parameters from my testI can see similar difference as before. Note I don’t care about parameters too much. In theory though those two builds should work similar, modulo some driver overhead difference in WSL2/Windows case.
Windows:
WSL
Ah okay. If you explicitly set
MIOPEN_DEBUG_CONV_DIRECT_NAIVE_CONV_FWD=1, it’ll overrideMIOPEN_DEBUG_CONV_DIRECT_NAIVE_CONV_FWD=0from the wheel’s__init.py__. Perhaps give that a try and see if you get any TDR’s on those larger resolutions?No idea just yet. I’m speculating that its a matter of tuning and perhaps doing a tuning run by following https://github.com/ROCm/hipBLASLt/blob/develop/docs/how-to/how-to-use-hipblaslt-tuning-utility.rst might help.
Yes, it’s with
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1, without it it fails withUserWarning: Flash Efficient attention on Current AMD GPU is still experimental. Enable it with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1.The WSL2 value is consistent, and faster across the runs, Windows one has more variation, it’s most of the time ~28 TFLOPs, but peak values can achieve 33 TFLOPs. I can gather some statistic from the runs, but not sure it will tell us anything useful, it’s probably power management difference or something.
@jammm: The comma is missing at the end, preprocessor joins “gfx1201” with “gfx950” making check fail.
This will fix that. (also other similar code is affected)
Great, thanks for checking that, yeah the gap is quite apparent. One last thing I’d like you to try - what’s the result of
torch.backends.cuda.preferred_blas_library()in wsl and windows native? Is it<_BlasBackend.Cublaslt: 2>or something else?@jammm: ComfyUI with default image generation workflow.
TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 python main.py --use-pytorch-cross-attentionfor trition case, which is the fastest currently on WSL2.You need to build torchvision too. torchaudio is not needed, it will complain, but not needed.
The CLR gfx1201 enablement patch: https://github.com/jammm/TheRock/commit/58cdbd09bd5c051a539b5d89fd8b9308994231be#diff-8aabe66220df2e98c07eec6d8515263399955f00a650e136142b80f7806b5178
@kasper93 , the CLR component used by ROCm on Windows is still missing a patch for gfx1201 support. @jammm and I got initial Windows builds for gfx1201 working with @stellaraccident help yesterday.