Tensile: [Bug]: Rocblas errors out on fp16 gemm operations

Describe the bug

rocblas returns rocblas_status_internal_error when rocblas_hgemm is called on mi100:

rocBLAS error: Tensile solution found, but exception thrown for { a_type: "f16_r", b_type: "f16_r", c_type: "f16_r", d_type: "f16_r", compute_type: "f16_r", transA: 'N', transB: 'N', M: 256, N: 256, K: 64, alpha: 1, row_stride_a: 1, col_stride_a: 256, row_stride_b: 1, col_stride_b: 128, row_stride_c: 1, col_stride_c: 256, row_stride_d: 1, col_stride_d: 256, beta: 0, batch_count: 1, strided_batch: true, stride_a: 0, stride_b: 0, stride_c: 0, stride_d: 0, atomics_mode: atomics_allowed }
Alpha value 7.21875 doesn't match that set in problem: 1
This message will be only be displayed once, unless the ROCBLAS_VERBOSE_TENSILE_ERROR environment variable is set.

To Reproduce

The issue can be repoduced with rocblas-bench: rocblas-bench -f gemm -r h -m 256 -n 256 -k 64 --transposeA N --transposeB N

Expected behavior

rocblas_hgemm should work for any valid input

Environment

Hardware description
CPU MI100
GPU EPYC 7402P
Software version
rocm-core v6.2.1
rocblas Self compiled @rocm-6.2.2 tag

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17

Commits related to this issue

Most upvoted comments

hmm that reaks of a compiler bug, maybe we should file against llvm, using f16c instructions should have no effect here.

Since this issue is now doubly confirmed to be fixed in rocm 6.3 (or whatever the next version will be called) im closeing it.

Hello @lubosz, thank you for reaching out. It appears that this issue may be specific to Arch Linux. Unfortunately, Arch Linux is not included in the official ROCm compatibility matrix, and the “official” ROCm package on Arch Linux is not maintained by AMD, so we can’t guarantee its functionality. You might consider submitting this issue to the Arch Linux community or trying the ROCm Docker image as an alternative.

Your results are quite strange I configured rocblas with:

cmake
    -B build
    -S .
    -D CMAKE_BUILD_TYPE=None
    -D CMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc
    -D CMAKE_TOOLCHAIN_FILE=toolchain-linux.cmake
    -D CMAKE_CXX_FLAGS="${CXXFLAGS} -fcf-protection=none"
    -D CMAKE_INSTALL_PREFIX=/opt/rocm
    -D CMAKE_PREFIX_PATH=/opt/rocm/llvm/lib/cmake/llvm
    -D amd_comgr_DIR=/opt/rocm/lib/cmake/amd_comgr
    -D BUILD_WITH_TENSILE=ON
    -D AMDGPU_TARGETS="gfx908:xnack-;gfx1030"
    -D Tensile_LIBRARY_FORMAT=msgpack
    -D Tensile_TEST_LOCAL_PATH="$srcdir/$_tensile_dir"

besides targets this is the same as the archlinux PKGBUILD.

i am on archlinux using tensile 013a4b7cbe31146925f55c62149190c45a561640 and rocblas at https://github.com/ROCm/rocBLAS/commit/a203ccb76e3e15c47e9617d7761e8c0f9081e6f5 on arch linux and this works fine here, but i will reopen the bug for your sake for now.