ROCm: [Issue]: JAX kernel generation error

Problem Description

Problem description

I am trying to run ColabFold, a scientific machine learning tool built on top of AlphaFold, to predict protein structure given a sequence of amino-acids. The following kernel generation problem arises on very large problem instances where a single GCD’s memory is not enough.

AlphaFold uses JAX/XLA to offload operations to GPUs.

:3:hip_module.cpp           :74  : 305270480312 us: [pid:312732 tid:0x15533460c740]  hipModuleGetFunction ( 0x7ffd650e5788, 0x559d78ba0e70, Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_SIA1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_U64SL1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 ) 
:1:hip_code_object.cpp      :624 : 305270480327 us: [pid:312732 tid:0x15533460c740] Cannot find the function: Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_SIA1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_U64SL1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 
:1:hip_module.cpp           :84  : 305270480339 us: [pid:312732 tid:0x15533460c740] Cannot find the function: Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_SIA1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_U64SL1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 for module: 0x78ba0e70
:3:hip_module.cpp           :85  : 305270480350 us: [pid:312732 tid:0x15533460c740] hipModuleGetFunction: Returned hipErrorNotFound : 
:3:hip_error.cpp            :36  : 305270480362 us: [pid:312732 tid:0x15533460c740]  hipGetLastError (  ) 
:3:hip_module.cpp           :74  : 305270480374 us: [pid:312732 tid:0x15533460c740]  hipModuleGetFunction ( 0x7ffd650e5788, 0x559d7aa260b0, Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_SIA1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_U64SL1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 ) 
:3:hip_module.cpp           :88  : 305270480397 us: [pid:312732 tid:0x15533460c740] hipModuleGetFunction: Returned hipSuccess : 
:3:hip_module.cpp           :470 : 305270480415 us: [pid:312732 tid:0x15533460c740]  hipExtModuleLaunchKernel ( 0x0x559d7d771340, 768, 66, 1, 256, 1, 1, 0, stream:0x559d688e6a50, char array:<null>, 0x7ffd650e57d0, event:0, event:0, 0 ) 
:3:rocvirtual.cpp           :798 : 305270480430 us: [pid:312732 tid:0x15533460c740] Arg0:  Tensor2dSizeA = val:110592
:3:rocvirtual.cpp           :798 : 305270480442 us: [pid:312732 tid:0x15533460c740] Arg1:  Tensor2dSizeB = val:2427264
:3:rocvirtual.cpp           :798 : 305270480453 us: [pid:312732 tid:0x15533460c740] Arg2:  AddressD = val:23372033148928
:3:rocvirtual.cpp           :798 : 305270480465 us: [pid:312732 tid:0x15533460c740] Arg3:  AddressC = val:23372033148928
--
:3:hip_graph.cpp            :1024: 305757611363 us: [pid:312732 tid:0x15533460c740]  hipGraphAddKernelNode ( 0x559d777c1cb0, 0x559d7f33a2c0, 0x7ffd650ed648, 1, 0x7ffd650ed2c0 ) 
:3:hip_graph_internal.cpp   :129 : 305757611376 us: [pid:312732 tid:0x15533460c740] [hipGraph] Add KernelNode(0x559d887c3bf0)
:3:hip_graph.cpp            :1036: 305757611388 us: [pid:312732 tid:0x15533460c740] hipGraphAddKernelNode: Returned hipSuccess : 
:3:hip_graph.cpp            :1024: 305757611403 us: [pid:312732 tid:0x15533460c740]  hipGraphAddKernelNode ( 0x559d777c1cb8, 0x559d7f33a2c0, 0x7ffd650ed648, 1, 0x7ffd650ed2c0 ) 
:3:hip_graph_internal.cpp   :129 : 305757611426 us: [pid:312732 tid:0x15533460c740] [hipGraph] Add KernelNode(0x559d83e52510)
:3:hip_graph.cpp            :1036: 305757611439 us: [pid:312732 tid:0x15533460c740] hipGraphAddKernelNode: Returned hipSuccess : 
:3:hip_graph.cpp            :1024: 305757611454 us: [pid:312732 tid:0x15533460c740]  hipGraphAddKernelNode ( 0x559d777c1cc0, 0x559d7f33a2c0, 0x7ffd650ed648, 1, 0x7ffd650ed2c0 ) 
:3:hip_graph.cpp            :1036: 305757611467 us: [pid:312732 tid:0x15533460c740] hipGraphAddKernelNode: Returned hipErrorInvalidConfiguration : 
E0904 10:53:24.344123  312732 pjrt_stream_executor_client.cc:2826] Execution of replica 0 failed: INTERNAL: Failed to add kernel node to a HIP graph: hipError_t(9)

In previous work, we worked with AMD devs to implement Managed Memory support (https://github.com/ROCm/ROCm/issues/3364) to run these large problem instances in the first place.

Steps to reproduce

The software has been built in a container: singularity pull colabfold.sif docker://quay.io/dipietrantonio/colabfold:v8

Here is the Dockerfile: https://github.com/PawseySC/pawsey-containers/blob/cdp-colabfold/colabfold/colabfold.dockerfile

Input file: https://github.com/user-attachments/files/16229258/1IH7.7.a3m.txt (remove the .txt extension)

We want the software to run on a MI250X, which are the GPUs powering the Setonix supercomputer at Pawsey Supercomputing Research Centre. We know that the same test case runs on a MI300 because it has enough memory to hold the entire computation in its GPU memory.

Script to run the test case:

A3M=1IH7.7.a3m # input is a multiple sequence alignment 
OUT=$MYSCRATCH/colabfold-out-1IH7.7 # name of the output directory 

# define colabfold container 
containerImage=/path/to/colabfold.sif


export SINGULARITYENV_XLA_PYTHON_CLIENT_PREALLOCATE=false
export SINGULARITYENV_TF_FORCE_UNIFIED_MEMORY="1"
export SINGULARITYENV_XLA_PYTHON_CLIENT_MEM_FRACTION="4.0"
export SINGULARITYENV_XLA_PYTHON_CLIENT_ALLOCATOR="platform"
export SINGULARITYENV_TF_FORCE_GPU_ALLOW_GROWTH="true"

openssl s_client -connect api.colabfold.com:443 

# Run colab container
singularity exec --rocm  $containerImage \
        colabfold_batch --num-recycle 5 --pair-mode unpaired --model-type alphafold2_multimer_v3 --num-models 3 $A3M $OUT

Operating System

Ubuntu 22 (in a container)

CPU

AMD Milan (zen3) CPU

GPU

MI250X

ROCm Version

ROCm 6.2.2

ROCm Component

No response

Steps to Reproduce

No response

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

No response

Additional Information

This ticket is a continuation of https://github.com/ROCm/ROCm/issues/3364

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 20 (1 by maintainers)

Most upvoted comments

@psanal35 Thank you for the tip! Disabling Triton GEMM has solved the reported issue. The software is currently running past the point where it would previously fail. So we can consider this issue solved.

Thank you very much for your help, and for the help of everyone involved in this work!

Hi @SarahBeecroft, yes, I have tested on an MI200 series.

@psanal35 Just checking- were you using MI250 or MI300? It’s not an issue on the MI300s, just 200 series.

I have got an update on my latest experiment with ColabFold, that is, running it on a smaller test case. The job times out after 1 hour (I had set 1 hour time limit), so it progresses further than the other one crashing after 10 minutes. But looking at the logfile, (grep -i error colabfold_batch_20264283_4294967294.err -A 7 -B 7) I see the following messages, which as very similar to the other case.

--
:3:hip_device_runtime.cpp   :636 : 3700546381595 us: [pid:1688798 tid:0x14bcd9061840]  hipGetDevice ( 0x7fff465e7ff4 ) 
:3:hip_device_runtime.cpp   :644 : 3700546381606 us: [pid:1688798 tid:0x14bcd9061840] hipGetDevice: Returned hipSuccess : 
:3:hip_device.cpp           :635 : 3700546381618 us: [pid:1688798 tid:0x14bcd9061840]  hipGetDevicePropertiesR0600 ( 0x7fff465e7ff8, 0 ) 
:3:hip_device.cpp           :637 : 3700546381629 us: [pid:1688798 tid:0x14bcd9061840] hipGetDevicePropertiesR0600: Returned hipSuccess : 
:3:hip_module.cpp           :74  : 3700546381656 us: [pid:1688798 tid:0x14bcd9061840]  hipModuleGetFunction ( 0x7fff465e86a8, 0x55b982caa730, Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_PKA0_SIA1_SLW1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 ) 
:1:hip_code_object.cpp      :1006: 3700546381670 us: [pid:1688798 tid:0x14bcd9061840] Cannot find the function: Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_PKA0_SIA1_SLW1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 
:1:hip_module.cpp           :84  : 3700546381681 us: [pid:1688798 tid:0x14bcd9061840] Cannot find the function: Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_PKA0_SIA1_SLW1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 for module: 0x82caa730
:3:hip_module.cpp           :85  : 3700546381692 us: [pid:1688798 tid:0x14bcd9061840] hipModuleGetFunction: Returned hipErrorNotFound : 
:3:hip_error.cpp            :36  : 3700546381712 us: [pid:1688798 tid:0x14bcd9061840]  hipGetLastError (  ) 
:3:hip_module.cpp           :74  : 3700546381725 us: [pid:1688798 tid:0x14bcd9061840]  hipModuleGetFunction ( 0x7fff465e86a8, 0x55b983f32ce0, Cijk_Ailk_Bljk_SB_MT128x96x16_MI32x32x2x1_SN_1LDSB0_APM1_ABV0_ACED0_AF0EM1_AF1EM1_AMAS0_ASE_ASGT_ASLT_ASM_ASAE01_ASCE01_ASEM1_AAC0_BL1_BS1_CLR0_DTLA0_DTLB0_DTVA0_DTVB0_DVO0_ETSP_EPS1_ELFLR0_EMLL0_FSSC10_FL0_GLVWA2_GLVWB2_GRCGA1_GRCGB1_GRPM1_GRVW2_GSU1_GSUASB_GLS0_ISA90a_IU1_K1_KLA_LBSPPA0_LBSPPB128_LPA0_LPB4_LDL1_LRVW4_LWPMn1_LDW0_FMA_MIAV1_MDA2_MO40_MMFGLC_MKFGSU256_NTA0_NTB0_NTC0_NTD0_NEPBS0_NLCA1_NLCB1_ONLL1_OPLV0_PK0_PAP0_PGR2_PLR9_PKA0_SIA1_SLW1_SS0_SU0_SUM0_SUS0_SCIUI1_SPO0_SRVW0_SSO4_SVW4_SNLL0_TSGRA0_TSGRB0_TT1_96_TLDS1_UMLDSA0_UMLDSB1_USFGROn1_VAW1_VSn1_VW1_VWB1_VFLRP0_WSGRA0_WSGRB0_WS64_WG128_2_1_WGM15 ) 
:3:hip_module.cpp           :88  : 3700546381739 us: [pid:1688798 tid:0x14bcd9061840] hipModuleGetFunction: Returned hipSuccess : 
:3:hip_module.cpp           :466 : 3700546381756 us: [pid:1688798 tid:0x14bcd9061840]  hipExtModuleLaunchKernel ( 0x0x55b988ba6fb0, 1280, 19, 1, 256, 1, 1, 0, stream:0x55b96dab18e0, char array:<null>, 0x7fff465e86f0, event:0, event:0, 0 ) 
:3:rocvirtual.cpp           :799 : 3700546381770 us: [pid:1688798 tid:0x14bcd9061840] Arg0:  Tensor2dSizeA = val:221184
:3:rocvirtual.cpp           :799 : 3700546381781 us: [pid:1688798 tid:0x14bcd9061840] Arg1:  Tensor2dSizeB = val:693504
:3:rocvirtual.cpp           :799 : 3700546381792 us: [pid:1688798 tid:0x14bcd9061840] Arg2:  AddressD = val:22722809858048
:3:rocvirtual.cpp           :799 : 3700546381803 us: [pid:1688798 tid:0x14bcd9061840] Arg3:  AddressC = val:22722809858048

However, the log does not present the message hipGraphAddKernelNode: Returned hipErrorInvalidConfiguration : which is present in the crash log of the bigger test case.

@al-rigazzi Sorry for late reply, I was following up with one of our colleagues and he said that he is looking into it and will update soon he has some results.

Hello @Ruturaj4 Can you please take a look at this?

Hi @dipietrantonio, thanks for reporting this. I’ve notified the team members that worked on https://github.com/ROCm/ROCm/issues/3364 and they are looking into this.