TheRock: [Windows]: HIPRTC_ERROR_COMPILATION on gfx1201

Problem Description

Not sure if I missed something or we are still is lacking gfx1201 support.

At least test_conv_transpose2d test says Unsupported PAL device with ASIC revision #62 which is something. But the other test_conv_cudnn_nhwc_support, just gives generic compilation error without much more info. (See results in additional info section)

I did default build with cmake -GNinja .. -DCMAKE_BUILD_TYPE=RelWithDebInfo -DTHEROCK_AMDGPU_FAMILIES=gfx1201 -DTHEROCK_AMDGPU_WINDOWS_INTEROP_DIR="E:\dev\amdgpu-windows-interop"

Also interesting is that when MIOpenDriver.exe is run as part of the test, it gives different result. I don’t know how it works, maybe you pass things in environment variable, but

Running MIOpenDriver.exe conv -n 1 -c 8 -H 5 -W 5 -k 4 -y 3 -x 3 -p 1 -q 1 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 2 -t 1 standalone crashes after it fails to find device. I would suggest just aborting/exiting earlier if such condition happens, instead of crashing later.

[0x0]   MIOpen!miopen::Handle::GetDeviceName+0xf   0x59f17fe890   0x7ff875940017   
[0x1]   MIOpen!miopen::solver::conv::GemmFwdBase::IsApplicable+0xa7   0x59f17fe8e0   0x7ff87594363f   
[0x2]   MIOpen!miopen::solver::conv::GemmFwd1x1_0_1::IsApplicable+0xf   0x59f17fead0   0x7ff875b9128b   
[0x3]   MIOpen!miopen::solver::SolverContainer<miopen::solver::conv::GemmFwd1x1_0_1,miopen::solver::conv::GemmFwd1x1_0_1_int8,miopen::solver::conv::GemmFwd1x1_0_2,miopen::solver::conv::GemmFwdRest,miopen::solver::conv::GemmBwd1x1_stride1,miopen::solver::conv::GemmBwd1x1_stride2,miopen::solver::conv::GemmBwdRest,miopen::solver::conv::GemmWrw1x1_stride1,miopen::solver::conv::GemmWrwUniversal>::GetWorkspaceSizes<miopen::ExecutionContext,miopen::conv::ProblemDescription>::<lambda_1>::operator()<miopen::solver::conv::GemmFwd1x1_0_1>+0xdb   0x59f17feb10   0x7ff875b910de   
[0x4]   MIOpen!miopen::each_args<`lambda at E:\dev\TheRock\ml-libs\MIOpen\src\include\miopen\find_solution.hpp:403:13',miopen::solver::conv::GemmFwd1x1_0_1,miopen::solver::conv::GemmFwd1x1_0_1_int8,miopen::solver::conv::GemmFwd1x1_0_2,miopen::solver::conv::GemmFwdRest,miopen::solver::conv::GemmBwd1x1_stride1,miopen::solver::conv::GemmBwd1x1_stride2,miopen::solver::conv::GemmBwdRest,miopen::solver::conv::GemmWrw1x1_stride1,miopen::solver::conv::GemmWrwUniversal>+0x1e   0x59f17fec90   0x7ff875b84c63   
[0x5]   MIOpen!miopen::solver::SolverContainer<miopen::solver::conv::GemmFwd1x1_0_1,miopen::solver::conv::GemmFwd1x1_0_1_int8,miopen::solver::conv::GemmFwd1x1_0_2,miopen::solver::conv::GemmFwdRest,miopen::solver::conv::GemmBwd1x1_stride1,miopen::solver::conv::GemmBwd1x1_stride2,miopen::solver::conv::GemmBwdRest,miopen::solver::conv::GemmWrw1x1_stride1,miopen::solver::conv::GemmWrwUniversal>::GetWorkspaceSizes<miopen::ExecutionContext,miopen::conv::ProblemDescription>+0x113   0x59f17fed10   0x7ff875b7fa70   
[0x6]   MIOpen!AllGemmWorkspaceSize+0x20   0x59f17fee30   0x7ff874d0c864   
[0x7]   MIOpen!miopen::`anonymous namespace'::GetWorkSpaceSizeGEMM+0x94   0x59f17fee70   0x7ff874d0b969   
[0x8]   MIOpen!miopen::ConvolutionDescriptor::GetWorkSpaceSize+0x959   0x59f17feed0   0x7ff874d23a4f   
[0x9]   MIOpen!miopenConvolutionBackwardDataGetWorkSpaceSize::<lambda_1>::operator()+0x2cf   (Inline Function)   (Inline Function)   
[0xa]   MIOpen!miopen::try_+0x2cf   (Inline Function)   (Inline Function)   
[0xb]   MIOpen!miopenConvolutionBackwardDataGetWorkSpaceSize+0x2ff   0x59f17ff090   0x7ff6b254b3df   
[0xc]   MIOpenDriver!ConvDriver<float,float>::AllocateBuffersAndCopy+0xb0f   0x59f17ff7b0   0x7ff6b29cde26   
[0xd]   MIOpenDriver!main+0x206   0x59f17ffb90   0x7ff6b346a9c4   
[0xe]   MIOpenDriver!invoke_main+0x22   (Inline Function)   (Inline Function)   
[0xf]   MIOpenDriver!__scrt_common_main_seh+0x10c   0x59f17ffc80   0x7ff959b5e8d7   
[0x10]   KERNEL32!BaseThreadInitThunk+0x17   0x59f17ffcc0   0x7ff95abdc5dc   
[0x11]   ntdll!RtlUserThreadStart+0x2c   0x59f17ffcf0   0x0   

Log before the above crash

:1:paldevice.cpp            :915 : 176315778463 us: [pid:20520 tid: 0x54632] Unsupported PAL device with ASIC revision #62
:3:comgrctx.cpp             :33  : 176315778692 us: [pid:20520 tid: 0x54632] Loading COMGR library.
:3:comgrctx.cpp             :128 : 176315778747 us: [pid:20520 tid: 0x54632] Loaded COMGR library version 3.0.
:3:hip_context.cpp          :54  : 176315778860 us: [pid:20520 tid: 0x54632] Direct Dispatch: 0
:3:hip_stream.cpp           :281 : 176315778887 us: [pid:20520 tid: 0x54632]  hipStreamCreate ( 00000059F17FFA58 )
:3:hip_stream.cpp           :281 : 176315778909 us: [pid:20520 tid: 0x54632] hipStreamCreate: Returned hipErrorNoDevice :
:3:hip_context.cpp          :166 : 176315778941 us: [pid:20520 tid: 0x54632]  hipInit ( 0 )
:3:hip_context.cpp          :166 : 176315778959 us: [pid:20520 tid: 0x54632] hipInit: Returned hipErrorNoDevice :
:3:hip_device_runtime.cpp   :671 : 176315778980 us: [pid:20520 tid: 0x54632]  hipGetDevice ( 00000059F17FF920 )
:3:hip_device_runtime.cpp   :671 : 176315778995 us: [pid:20520 tid: 0x54632] hipGetDevice: Returned hipErrorNoDevice :
MIOpen Error: E:/dev/TheRock/ml-libs/MIOpen/src/hip/handlehip.cpp:150: No device
MIOpen(HIP): Info [] MIOPEN_FIND_MODE = NORMAL(1)
MIOpen(HIP): Info [AmdRocmMetadataVersionDetect] ROCm MD version AMDHSA_COv3, HIP version 6.5.25204, MIOpen version 3.4.0.204b36923
MIOpen(HIP): Info2 [GetWorkSpaceSize]
MIOpen(HIP): Info2 [GetWorkspaceSizes] fft: Not applicable
<crash>

Operating System

10.0.26100

CPU

AMD Ryzen 9 7950X 16-Core Processor

GPU

AMD Radeon RX 9070 XT

ROCm Version

TheRock a742718b

ROCm Component

HIP

Steps to Reproduce

Run python -m pytest -v smoke-tests on gfx1201

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

Test results

smoke-tests/pytorch_smoke_test.py::TestROCmAvailability::test_rocm_available PASSED                                                                                                                                                                                                                                   [  9%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_multiplication PASSED                                                                                                                                                                                                                            [ 18%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_batch_matrix_multiplication PASSED                                                                                                                                                                                                                      [ 27%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_multiplication_at_operator PASSED                                                                                                                                                                                                                [ 36%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_elementwise_multiplication PASSED                                                                                                                                                                                                                       [ 45%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_transpose PASSED                                                                                                                                                                                                                                        [ 54%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_dot_product PASSED                                                                                                                                                                                                                                      [ 63%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_vector_multiplication PASSED                                                                                                                                                                                                                     [ 72%]
smoke-tests/pytorch_smoke_test.py::TestMatrixOperations::test_matrix_multiplication_matmul PASSED                                                                                                                                                                                                                     [ 81%]
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_transpose2d FAILED                                                                                                                                                                                                                                     [ 90%]
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_cudnn_nhwc_support FAILED                                                                                                                                                                                                                              [100%]
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_transpose2d

Interesting part of log:

MIOpen(HIP): Info2 [LoadBinary] Unable to load binary for: "naive_conv.cpp.obj"; args:  -DMIOPEN_USE_FP16=0 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=1 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201
MIOpen(HIP): Info [PrintVersion] HIPRTC v.9.0
MIOpen(HIP): Trace [BuildHip] HIPRTC compile ROCm path: E:\dev\TheRock\build\dist\rocm\
:1:paldevice.cpp            :915 : 175388348189 us: [pid:5720 tid: 0x56464] Unsupported PAL device with ASIC revision #62
:3:comgrctx.cpp             :33  : 175388348340 us: [pid:5720 tid: 0x56464] Loading COMGR library.
:3:comgrctx.cpp             :128 : 175388348376 us: [pid:5720 tid: 0x56464] Loaded COMGR library version 3.0.
:3:hip_context.cpp          :54  : 175388348456 us: [pid:5720 tid: 0x56464] Direct Dispatch: 0
:3:hip_device_runtime.cpp   :671 : 175388348470 us: [pid:5720 tid: 0x56464]  hipGetDevice ( 000000CEBDFE70D4 )
:3:hip_device_runtime.cpp   :671 : 175388348476 us: [pid:5720 tid: 0x56464] hipGetDevice: Returned hipErrorNoDevice :
MIOpen(HIP): Error [Compile] 'hiprtcCompileProgram(prog.get(), c_options.size(), c_options.data())' naive_conv.cpp: HIPRTC_ERROR_COMPILATION (6)
MIOpen(HIP): Error [BuildHip] HIPRTC status = HIPRTC_ERROR_COMPILATION (6), source file: naive_conv.cpp
MIOpen(HIP): Warning [BuildHip] warning: HIPRTC error log empty
smoke-tests/pytorch_smoke_test.py::TestConvolutions::test_conv_cudnn_nhwc_support

Interesting part of log, note the difference from previous one, there are no logs from HIP at all:

MIOpen(HIP): Trace [BuildHip] HIPRTC compile ROCm path: E:\dev\TheRock\build\dist\rocm\
MIOpen(HIP): Error [Compile] 'hiprtcCompileProgram(prog.get(), c_options.size(), c_options.data())' naive_conv.cpp: HIPRTC_ERROR_COMPILATION (6)
MIOpen(HIP): Error [BuildHip] HIPRTC status = HIPRTC_ERROR_COMPILATION (6), source file: naive_conv.cpp
MIOpen(HIP): Warning [BuildHip] warning: HIPRTC error log empty
MIOpen(HIP): Info2 [StoreRecord] Trying to store record at key 16-1-1-3x3-8-1-1-1-1x1-2x1-1x1-0-NCHW-FP32-F in cache for file "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [StoreRecordUnsafe] Storing record: 16-1-1-3x3-8-1-1-1-1x1-2x1-1x1-0-NCHW-FP32-F
MIOpen(HIP): Info2 [FindRecordUnsafe] Looking for key 16-1-1-3x3-8-1-1-1-1x1-2x1-1x1-0-NCHW-FP32-F in file "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [UpdateDbModificationTime] Updating db modification time for "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [ValidateUnsafe] DB file is newer than cache: 175388367555100, 175388364048400
MIOpen(HIP): Info2 [UpdateDbModificationTime] Updating db modification time for "C:\\Users\\Kacper\\.miopen\\db\\gfx1201_32.HIP.3_4_0_204b36923.ufdb.txt"
MIOpen(HIP): Info2 [Measure] Db::StoreRecord time: 0.6867 ms
MIOpen Error: E:/dev/TheRock/ml-libs/MIOpen/src/hipoc/hipoc_program.cpp:299: Code object build failed. Source: naive_conv.cpp

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 77 (40 by maintainers)

Commits related to this issue

Most upvoted comments

I did some hipblaslt tuning, and it looks like it’s absolutely necessary if you want to get reasonable performance. Some rough comparison.

before tuning after tuning
SD1.5 (512x512) 11.16it/s 22.02it/s
SDXL (768x1344) 1.82it/s 4.75it/s
conv2d-f16.py 0.565ms 0.293ms

@jammm at some point, I would like a standalone script that runs some performance diagnostics and runs some key benchmarks. We’ve got this embedded in our qa procedures, but every time we go to support this is the field, we have a bunch of back and forths like this one. I’d love to just have first level performance debugging to have step 1 be “run this script and paste results”.

If we had that, I daresay it would short circuit half of the field support issues I see on numerous forums.

Maybe: validate_amd_pytorch_goes_vroom.py

Heads up that if MIOpen is reporting that it’s using the Winograd solver, from the above:

miopenSp3AsmConvFury_v2_4_1_gfx11_1536vgprs_fp16_fp16acc_f2x3_c32_stride1

This means MIOpen isn’t using explicit GEMM for this. If not using explicit GEMM then there is no use of Im2col or a GEMM from rocBLAS or hipBLASLt so no tuning of those will help in this case. Though in general MIOpen is going to try and pick between multiple solver choices which on this hardware likely includes:

  • Direct (naive) solvers: these are slower usually though not for some small shapes but are very generally applicable
  • Winograd solvers: uses the winograd method of doing convolution so no GEMM at all but limited applicability
  • explicit GEMM solvers: these will convert the convolution to a GEMM using Im2Col and use a GEMM to do the work so these will depend on ensuring hipBLASLt is used and is tuned

MIOpen also has a number of implicit GEMM based solvers but not many of these are available in Windows yet but we’re working on it.

I did some hipblaslt tuning, and it looks like it’s absolutely necessary if you want to get reasonable performance. Some rough comparison. before tuning after tuning SD1.5 (512x512) 11.16it/s 22.02it/s SDXL (768x1344) 1.82it/s 4.75it/s conv2d-f16.py 0.565ms 0.293ms

Indeed, this confirms my suspicions that tuning would be needed. I just don’t have a gfx1201 to do tuning with. Feel free to create a PR for this tuning and we could incorporate this tuning in TheRock for the time being.

Happy to land it in hipblaslt as well.

For me it’s

With MIOPEN_GEMM_ENFORCE_BACKEND=1: Im2d2Col_v2 into Cijk_Ailk_Bljk_HHS_BH_MT64x32x8_SN_1LDSB0_AMAS2_BL0_BS1_EPS0_GLVWA2_GLVWB1_GRVW2_GSU1_GSUASB_ISA000_IU1_K1_KLS_LBSPPA0_LBSPPB0_LPA0_LPB0_LRVW2_MIAV0_MMFSC_NLCA1_NLCB1_PGR0_PLR1_SIA1_SS0_SU32_SUM0_SUS256_SVW4_TT4_2_TLDS0_UMLDSA0_UMLDSB0_USFGRO0_VAW2_VS1_VW2_VWB2_WSGRA0_WSGRB0_WS64_WG16_16_1_WGM8

With MIOPEN_GEMM_ENFORCE_BACKEND=5: Im2d2Col_v2 into Cijk_Ailk_Bljk_HHS_BH_UserArgs_MT16x16x64_MI16x16x1_SN_LDSB0_AFC0_AFEM1_AFEM1_ASEM1_CLR1_CADS0_DTVA0_DTVB0_EPS0_FDSI0_GRPM1_GRVWA8_GRVWB8_GSUAMB_GLS0_ISA1201_IU1_K1_LBSPPA256_LBSPPB256_LBSPPM0_LPA16_LPB16_LPM0_LRVW8_LWPMn1_MIAV1_MIWT1_1_MO40_NTn1_NTA0_NTB0_NTC0_NTD0_NTM0_NEPBS0_NLCA1_NLCB1_ONLL1_PGR0_PLR0_PKA0_SIA3_SS0_SPO0_SRVW0_SSO0_SVW8_SK0_SKXCCM0_TLDS0_ULSGRO0_USL1_UIOFGRO0_USFGROn1_VSn1_VWA1_VWB1_WSGRA0_WSGRB0_WS32_WG16_2_1

and the difference difference in perf is significant. Notice that it has UserArgs, because hipblaslt has some gfx1201 tuning files in it.

BTW the perf I get on this conv2d script also seems better than gfx1201, oddly enough:

Yeah, on gfx1151, it uses https://github.com/ROCm/MIOpen/blob/66aec60520122a1972cb4914bc7a9c59ceede69d/src/solver/conv/conv_wino_fury_RxS.cpp for this convolution.

And looking at this https://github.com/ROCm/MIOpen/blob/66aec60520122a1972cb4914bc7a9c59ceede69d/src/solver/conv/conv_wino_fury_RxS.cpp#L254-L256 it’s seems to be only for gfx11… which well, maybe is true, or maybe is just not updated for newer arch. (there are winograd solvers for gfx9, gfx10, gfx11, not yet updated for gfx12 it seems)

Honestly those hardcoded and scattered around checks in various ROCm parts are getting quite tiresome to track.

I guess I should buy gfx950, which is blessed and tested target.

No idea just yet. I’m speculating that its a matter of tuning and perhaps doing a tuning run by following https://github.com/ROCm/hipBLASLt/blob/develop/docs/how-to/how-to-use-hipblaslt-tuning-utility.rst might help.

Maybe it will help with overall performance. But as I understand it should currently use the same tuning in both Native Windows and WSL2 case. So I suspect there is still some other config difference between those two.

One last thing I’d like you to try - what’s the result of torch.backends.cuda.preferred_blas_library() in wsl and windows native? Is it <_BlasBackend.Cublaslt: 2> or something else?

That’s our problem:

_BlasBackend.Cublas

And when I set TORCH_BLAS_PREFER_CUBLASLT=1 I get warning that it’s not supported.

[W525 01:12:06.000000000 Context.cpp:375] Warning: Attempting to use hipBLASLt on an unsupported architecture! Overriding blas backend to hipblas (function operator())
_BlasBackend.Cublas

EDIT:

Forgot to say, on WSL it’s using _BlasBackend.Cublaslt

Interestingly, I did see this message before I tried running the image build in comfyui, so probably need to solve it first:

That could be solved by pip install "numpy<2" but it shouldn’t be the cause of the TDR’s.

I’m not sure what’s causing your issue. Perhaps @kasper93 is right and the two ROCm’s are clashing against each other somehow. Can you try temporarily renaming the ROCm folder you installed yourself and try again perhaps?

Heads up that if MIOpen is reporting that it’s using the Winograd solver, from the above:

miopenSp3AsmConvFury_v2_4_1_gfx11_1536vgprs_fp16_fp16acc_f2x3_c32_stride1

This means MIOpen isn’t using explicit GEMM for this. If not using explicit GEMM then there is no use of Im2col or a GEMM from rocBLAS or hipBLASLt so no tuning of those will help in this case. Though in general MIOpen is going to try and pick between multiple solver choices which on this hardware likely includes:

* Direct (naive) solvers: these are slower usually though not for some small shapes but are very generally applicable

* Winograd solvers: uses the winograd method of doing convolution so no GEMM at all but limited applicability

* explicit GEMM solvers: these will convert the convolution to a GEMM using Im2Col and use a GEMM to do the work so these will depend on ensuring hipBLASLt is used and is tuned

MIOpen also has a number of implicit GEMM based solvers but not many of these are available in Windows yet but we’re working on it.

Thank you for insight. I’m focused only on gfx12 currently and none of Winograd solvers are supported yet. I’m messing around with GEMM and differences between rocBLAS or hipBLASLt. But indeed this Winograd kernel was a reason why @jammm didn’t see performance difference in workloads where I’ve seen it, because it was always using kernel from miopen directly.

@jammm @scottt : The performance difference I’m seeing is because rocBLAS is built without hipBLASLt. See BUILD_WITH_HIPBLASLT in rocBLAS.

That’s also why ROCBLAS_USE_HIPBLASLT doesn’t work to force it, and either way it would use hipBLASLt by default for this case. MIOpen always calls rocBLAS (it’s default backend for GEMM).

If I set MIOPEN_GEMM_ENFORCE_BACKEND=5 to force MIOpen to call hipBLASLt directly, I get the same performance on both builds in all test cases.

MIOPEN_GEMM_ENFORCE_BACKEND unset

3x3 conv (320->320, 64x64, fp16):
Time: 0.708ms

MIOPEN_GEMM_ENFORCE_BACKEND=5

3x3 conv (320->320, 64x64, fp16):
Time: 0.565ms

Hey @jammm im from the shark discord.

I think your post on X about the 60it/s is not quite right at least when I test it. It seems that comfy reports the wrong it/s for euler because the gpu still keeps going after then gen is already at 50/50 in the console. If you look at the UI with euler it takes a long time for the image to appear. When testing other samplers that are also as fast as euler like dpmpp_2m it takes no time for the image to appear and the overall time is the same. Meaning that euler reports as finished but in reality its not thats why the image takes so long to appear because its still being sampled.

I hear the audible coil whine still going after the image is already finished in the terminal.

Image

Hmm, there seem to be some GPU/CPU synchronization “fun”. On WSL2 it actually doesn’t change much, but on native videos, using --gpu-only which keeps everything in VRAM and run text encoders on GPU too. Makes significant difference on new pytorch 2.7. Not sure if this is “Windows” thing or some other factor going from 2.6 to 2.7 pytorch, because on WSL2 it’s 2.6.

WSL2 ROCm 6.4.1 (pytorch 2.6) torch-2.7.0a0+git7027b2a-cp312-cp312-win_amd64.whl SD1.5 / ComfyUI / --use-pytorch-cross-attention 10.84it/s 7.99it/s SD1.5 / ComfyUI / --use-pytorch-cross-attention / --gpu-only 11.16it/s 55.87it/s

Interesting. I also tried --gpu-only and now on the gfx1151 it goes to 70-80 it/s

Image

Hmm, there seem to be some GPU/CPU synchronization “fun”. On WSL2 it actually doesn’t change much, but on native windows, using --gpu-only which keeps everything in VRAM and run text encoders on GPU too. Makes significant difference on new pytorch 2.7. Not sure if this is “Windows” thing or some other factor going from 2.6 to 2.7 pytorch, because on WSL2 it’s 2.6.

WSL2 ROCm 6.4.1 (pytorch 2.6) torch-2.7.0a0+git7027b2a-cp312-cp312-win_amd64.whl
SD1.5 / ComfyUI / --use-pytorch-cross-attention 10.84it/s 7.99it/s
SD1.5 / ComfyUI / --use-pytorch-cross-attention / --gpu-only 11.16it/s 55.87it/s

FYI, I sent some fixes for ComfyUI, https://github.com/comfyanonymous/ComfyUI/pull/8289 . On WSL2 I get VAE speed up and this dreaded OOM on VAE Decode is no more. Though TDR is still there on huge sizes.

Note that this is not related to perf difference between builds discussed in this thread, it’s just making ComfyUI faster in general.

@jammm , besides investigating why our build of MIOpen does not use hipBLASLt, note that our first cut of validate_torch_vroom.py did not capture the perf difference that @kasper93 is seeing with v1-5-pruned-emaonly.safetensors 512x512.

One idea to improve the latter, is we do some Pytorch level profiling of end-to-end models runs like sdxl-base-1.0.py , capture the input parameters and shapes of the critical stages and measure those operations in validate-torch-perf.py.

@kasper93 , torch.nn.Conv2d calls MIOpen for AMD GPU builds. Enable MIOpen logging with its environment variables and comparing the logs of the slow and fast conv2d should give us a clue:

See the logs from the MIOpen below, there isn’t really much difference. The main thing, is that HIPBLASLT logs are missing from slow (Windows) build, [HIPBLASLT][72350][Api][rocblaslt_algo_get_heuristic_cpp] returnAlgoCount=1.

Notable difference is that at the beginning of the execution there are two rocblaslt_create calls in “fast” build and single call in "slow’ build, which may suggest that some component is not creating own handle in Windows case.

What I did initially is to benchmark some common primitives and this conv was one of the things that stood up when comparing both builds. There were other things, but one at the time…

I wanted to get hipblaslt-bench command, but on Windows build HIPBLASLT_LOG_MASK=32 doesn’t work for this test case, while it does on WSL2 build. Also ROCBLAS_USE_HIPBLASLT doesn’t seem to affect this case.

(logs stripped from clarity)

Windows:

MIOpen(HIP): Info [FindConvolution] miopenConvolutionFwdAlgoGEMM	0.762628	23592960
MIOpen(HIP): Info [FillFindReturnParameters] FW Chosen Algorithm: GemmFwdRest , 23592960, 0.762628
MIOpen(HIP): Command [LogCmdConvolution] MIOpenDriver.exe convfp16 -n 1 -c 320 -H 64 -W 64 -k 320 -y 3 -x 3 -p 1 -q 1 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1
MIOpen(HIP): Info [ConvolutionForward] algo = 0, workspace = 23592960
MIOpen(HIP): Info2 [GetInvoker] Returning an invoker for problem 320x64x64x3x3x320x64x64x1xNCHWxFP16x1x1x1x1x1x1x1xFxDefault and algorithm miopenConvolutionFwdAlgoGEMM
MIOpen(HIP): Info2 [GetKernels] 0 kernels for key: miopenIm2d2Col "c320i64_64w3_3p1_1s1_1d1_1t0"
MIOpen(HIP): Info2 [AddKernel] Key: miopenIm2Col "c320i64_64w3_3p1_1s1_1d1_1t0"
MIOpen(HIP): Info2 [SQLiteBase] Initializing system database file ""
MIOpen(HIP): Info [KernDb] database not present
MIOpen(HIP): Info2 [SQLiteBase] Initializing user database file "C:\\Users\\Kacper\\.miopen\\cache\\3.4.0.67f31ae2f\\gfx1201_32.ukdb"
MIOpen(HIP): Trace [Exec] 38588:PRAGMA journal_mode=WAL;
MIOpen(HIP): Trace [Exec] 38588:CREATE TABLE IF NOT EXISTS `kern_db` (`id` INTEGER PRIMARY KEY ASC,`kernel_name` TEXT NOT NULL,`kernel_args` TEXT NOT NULL,`kernel_blob` BLOB NOT NULL,`kernel_hash` TEXT NOT NULL,`uncompressed_size` INT NOT NULL);CREATE UNIQUE INDEX IF NOT EXISTS `idx_kern_db` ON kern_db(kernel_name, kernel_args);
MIOpen(HIP): Info2 [KernDb] Database created successfully
MIOpen(HIP): Trace [Exec] 38588:PRAGMA table_info(kern_db);
MIOpen(HIP): Info2 [LoadBinary] Loading binary for: "MIOpenIm2d2Col.cl.obj"; args:  -DLOCAL_MEM_SIZE=340 -DSTRIDE_GT_1=0 -DNUM_IM_BLKS_EQ_1=0 -DUSE_IM_OFF_GUARD=1 -DMIOPEN_USE_FP16=1 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=0 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201
MIOpen(HIP): Info2 [Prepare] SELECT kernel_blob, kernel_hash, uncompressed_size FROM kern_db WHERE (kernel_name = 'MIOpenIm2d2Col.cl.obj') AND (kernel_args = ' -DLOCAL_MEM_SIZE=340 -DSTRIDE_GT_1=0 -DNUM_IM_BLKS_EQ_1=0 -DUSE_IM_OFF_GUARD=1 -DMIOPEN_USE_FP16=1 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=0 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201');
MIOpen(HIP): Info2 [Measure] Db::FindRecord time: 0.2011 ms
MIOpen(HIP): Info2 [LoadBinary] Successfully loaded binary for: "MIOpenIm2d2Col.cl.obj"; args:  -DLOCAL_MEM_SIZE=340 -DSTRIDE_GT_1=0 -DNUM_IM_BLKS_EQ_1=0 -DUSE_IM_OFF_GUARD=1 -DMIOPEN_USE_FP16=1 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=0 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201
MIOpen(HIP): Info2 [run] kernel_name = Im2d2Col_v2, global_work_dim = { 1310720, 1, 1 }, local_work_dim = { 256, 1, 1 }
MIOpen(HIP): Info2 [CallGemm] gemm_desc: {isColMajor 0, transA 0, transB 0, m 320, n 4096, k 2880, lda 2880, ldb 4096, ldc 4096, batch_count 1, strideA 0, strideB 0, strideC 0, alpha 1, beta 0, dataType half, a_cast_type half, b_cast_type half} 
MIOpen(HIP): Info2 [CallGemm] rocBLAS

WSL2:

MIOpen(HIP): Info [FindConvolution] miopenConvolutionFwdAlgoGEMM	0.533991	23592960
MIOpen(HIP): Info [FillFindReturnParameters] FW Chosen Algorithm: GemmFwdRest , 23592960, 0.533991

MIOpen(HIP): Command [LogCmdConvolution] ./bin/MIOpenDriver convfp16 -n 1 -c 320 -H 64 -W 64 -k 320 -y 3 -x 3 -p 1 -q 1 -u 1 -v 1 -l 1 -j 1 -m conv -g 1 -F 1 -t 1
MIOpen(HIP): Info [ConvolutionForward] algo = 0, workspace = 23592960
MIOpen(HIP): Info2 [GetInvoker] Returning an invoker for problem 320x64x64x3x3x320x64x64x1xNCHWxFP16x1x1x1x1x1x1x1xFxDefault and algorithm miopenConvolutionFwdAlgoGEMM

MIOpen(HIP): Info2 [GetKernels] 0 kernels for key: miopenIm2d2Col "c320i64_64w3_3p1_1s1_1d1_1t0"
MIOpen(HIP): Info2 [AddKernel] Key: miopenIm2Col "c320i64_64w3_3p1_1s1_1d1_1t0"

MIOpen(HIP): Info2 [SQLiteBase] Initializing system database file ""
MIOpen(HIP): Info [KernDb] database not present
MIOpen(HIP): Info2 [SQLiteBase] Initializing user database file "/home/kacper/.cache/miopen/3.4.0.f10c6ed80-dirty/gfx1201_32.ukdb"
MIOpen(HIP): Trace [Exec] 139224243810432:PRAGMA journal_mode=WAL;
MIOpen(HIP): Trace [Exec] 139224243810432:CREATE TABLE IF NOT EXISTS `kern_db` (`id` INTEGER PRIMARY KEY ASC,`kernel_name` TEXT NOT NULL,`kernel_args` TEXT NOT NULL,`kernel_blob` BLOB NOT NULL,`kernel_hash` TEXT NOT NULL,`uncompressed_size` INT NOT NULL);CREATE UNIQUE INDEX IF NOT EXISTS `idx_kern_db` ON kern_db(kernel_name, kernel_args);
MIOpen(HIP): Info2 [KernDb] Database created successfully
MIOpen(HIP): Trace [Exec] 139224243810432:PRAGMA table_info(kern_db);
MIOpen(HIP): Info2 [LoadBinary] Loading binary for: "MIOpenIm2d2Col.cl.o"; args:  -DLOCAL_MEM_SIZE=340 -DSTRIDE_GT_1=0 -DNUM_IM_BLKS_EQ_1=0 -DUSE_IM_OFF_GUARD=1 -DMIOPEN_USE_FP16=1 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=0 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201
MIOpen(HIP): Info2 [Prepare] SELECT kernel_blob, kernel_hash, uncompressed_size FROM kern_db WHERE (kernel_name = 'MIOpenIm2d2Col.cl.o') AND (kernel_args = ' -DLOCAL_MEM_SIZE=340 -DSTRIDE_GT_1=0 -DNUM_IM_BLKS_EQ_1=0 -DUSE_IM_OFF_GUARD=1 -DMIOPEN_USE_FP16=1 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=0 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201');
MIOpen(HIP): Info2 [Measure] Db::FindRecord time: 0.160518 ms
MIOpen(HIP): Info2 [LoadBinary] Successfully loaded binary for: "MIOpenIm2d2Col.cl.o"; args:  -DLOCAL_MEM_SIZE=340 -DSTRIDE_GT_1=0 -DNUM_IM_BLKS_EQ_1=0 -DUSE_IM_OFF_GUARD=1 -DMIOPEN_USE_FP16=1 -DMIOPEN_USE_FP16x4=0 -DMIOPEN_USE_FP16x8=0 -DMIOPEN_USE_FP32=0 -DMIOPEN_USE_INT8=0 -DMIOPEN_USE_BFP16=0 -DMIOPEN_USE_INT32=0 -DMIOPEN_USE_RNE_BFLOAT16=1 -DMIOPEN_FP8_IEEE_EXPONENT_BIAS=0 -DMIOPEN_FP8_CLIPPING=1 -mcpu=gfx1201
MIOpen(HIP): Info2 [run] kernel_name = Im2d2Col_v2, global_work_dim = { 1310720, 1, 1 }, local_work_dim = { 256, 1, 1 }
MIOpen(HIP): Info2 [CallGemm] gemm_desc: {isColMajor 0, transA 0, transB 0, m 320, n 4096, k 2880, lda 2880, ldb 4096, ldc 4096, batch_count 1, strideA 0, strideB 0, strideC 0, alpha 1, beta 0, dataType half, a_cast_type half, b_cast_type half} 
MIOpen(HIP): Info2 [CallGemm] rocBLAS
[HIPBLASLT][72350][Api][rocblaslt_algo_get_heuristic_cpp] returnAlgoCount=1 

This validate_torch_vroom.py from @jammm measures (1) scaled dot-product attention (2) conv2d (3) GEMM throughput. It’d be interesting too see the numbers between your slow and fast builds as well.

I don’t see any meaningful difference in those tests.

But if I change conv2d to parameters from my test

    N, Cin, Cout = 1, 320, 320
    H, W = 64, 64
    kernel_size = 3
    stride = 1
    padding = 1

I can see similar difference as before. Note I don’t care about parameters too much. In theory though those two builds should work similar, modulo some driver overhead difference in WSL2/Windows case.

Windows:

Benchmarking Conv2d in FP16 ...
Average time: 0.83 ms
Average peak memory: 29.26 MB
Average throughput: 9117.10 GFLOP/s

WSL

Benchmarking Conv2d in FP16 ...
Average time: 0.62 ms
Average peak memory: 29.26 MB
Average throughput: 12198.45 GFLOP/s

Thanks for sharing. Where can I repro this? TDR’s are also bad tbh, but i can always remove that env var

In ComfyUI, 1536x2688 image feed into VAE Encode will fail. Though more regular sizes seems to not cause problem.

Ah okay. If you explicitly set MIOPEN_DEBUG_CONV_DIRECT_NAIVE_CONV_FWD=1, it’ll override MIOPEN_DEBUG_CONV_DIRECT_NAIVE_CONV_FWD=0 from the wheel’s __init.py__. Perhaps give that a try and see if you get any TDR’s on those larger resolutions?

SD1.5 / ComfyUI / TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 / --use-pytorch-cross-attention

Do you have idea why this one differs? aotriton seems to work (at least in standalone test) and hipblaslt is fine too, even tho looks like tuning for both may not be optimal. I can still reproduce a perf gap from WSL2 build, which seems to be unrelated to both. Do you have suggestions what can we test to diagnose this?

No idea just yet. I’m speculating that its a matter of tuning and perhaps doing a tuning run by following https://github.com/ROCm/hipBLASLt/blob/develop/docs/how-to/how-to-use-hipblaslt-tuning-utility.rst might help.

Interesting. In sdpa_bench.py, Is the peak memory usage mentioned in the script output the same for both windows/wsl? Also, run that script with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 for WSL (windows wheel automatically sets it)

Yes, it’s with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1, without it it fails with UserWarning: Flash Efficient attention on Current AMD GPU is still experimental. Enable it with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1.

Is the peak memory usage mentioned in the script output the same for both windows/wsl?

The WSL2 value is consistent, and faster across the runs, Windows one has more variation, it’s most of the time ~28 TFLOPs, but peak values can achieve 33 TFLOPs. I can gather some statistic from the runs, but not sure it will tell us anything useful, it’s probably power management difference or something.

Thanks! Hmm. I thought I had fixed that with this patch but it seems like that’s not the case jammm@2198660 i’ll look into this tomorrow

@jammm: The comma is missing at the end, preprocessor joins “gfx1201” with “gfx950” making check fail.

This will fix that. (also other similar code is affected)

diff --git a/aten/src/ATen/Context.cpp b/aten/src/ATen/Context.cpp
index d384b5e6b2..724c95edfd 100644
--- a/aten/src/ATen/Context.cpp
+++ b/aten/src/ATen/Context.cpp
@@ -359,7 +359,7 @@ at::BlasBackend Context::blasPreferredBackend() {
       static const std::vector<std::string> archs = {
           "gfx90a", "gfx942",
 #if ROCM_VERSION >= 60300
-          "gfx1100", "gfx1101", "gfx1150", "gfx1151", "gfx1200", "gfx1201"
+          "gfx1100", "gfx1101", "gfx1150", "gfx1151", "gfx1200", "gfx1201",
 #endif
 #if ROCM_VERSION >= 60500
           "gfx950"

Which pytorch wheel did you use for WSL?

The ones from https://repo.radeon.com/rocm/manylinux/rocm-rel-6.4.1/

torch-2.6.0+rocm6.4.1.git1ded221d-cp312-cp312-linux_x86_64.whl torchaudio-2.6.0+rocm6.4.1.gitd8831425-cp312-cp312-linux_x86_64.whl torchvision-0.21.0+rocm6.4.1.git4040d51f-cp312-cp312-linux_x86_64.whl pytorch_triton_rocm-3.2.0+rocm6.4.1.git6da9e660-cp312-cp312-linux_x86_64.whl

For windows native, was this the first time you used pytorch, or did you use any pytorch+rocm wheels before that? I’m wondering if the previous wheels left some cached kernels that might be conflicting here.

I did run before other wheel. I recreate whole venv each time I run different version though.

Now I also removed %userprofile%\.miopen to be sure it doesn’t use cache from there, but no difference.

For a more easily reproducible benchmark how about running

Yeah, sure. I didn’t let it run much, but it starts from the safe configurations and it’s clear that perf is completely different.

Windows

The best outcome was 12.4TFLOPS @ 2304x5120x1536 (MxNxK) (tried 15 shapes)

WSL2

The best outcome was 45.7TFLOPS @ 2304x5120x1536 (MxNxK) (tried 9 shapes)

Great, thanks for checking that, yeah the gap is quite apparent. One last thing I’d like you to try - what’s the result of torch.backends.cuda.preferred_blas_library() in wsl and windows native? Is it <_BlasBackend.Cublaslt: 2> or something else?

@jammm: ComfyUI with default image generation workflow. TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 python main.py --use-pytorch-cross-attention for trition case, which is the fastest currently on WSL2.

You need to build torchvision too. torchaudio is not needed, it will complain, but not needed.

@kasper93 , the CLR component used by ROCm on Windows is still missing a patch for gfx1201 support. @jammm and I got initial Windows builds for gfx1201 working with @stellaraccident help yesterday.