Tensile: [Bug]: rocBLAS error: Cannot read TensileLibrary.dat: No such file or directory

Describe the bug

Basically getting some form of this error, either rocBLAS error: Cannot read /opt/rocm-5.4.0/lib/rocblas/library/TensileLibrary.dat: Illegal seek or Cannot read TensileLibrary.dat: No such file or directory

To Reproduce

rocblas-dev (= 2.46.0.50400-72~22.04)

Steps to reproduce the behavior:

  1. Basically followed the guide on AMD Docs to install ROCm along with almost all usecases because I kept having errors with packages missing.
  2. Built wheel for onnxruntime using docker
  3. Trying to run the python application (roop) and getting this error

Expected behavior

No error?

Log-files

Aborted (core dumped)
(roop) hobi@hobi:~/roop$ ~python run.py --execution-provider rocm --execution-threads
Command '~python' not found, did you mean:
  command 'bpython' from deb bpython (0.22.1-2)
  command 'xpython' from deb xpython (0.12.5-1build1)
Try: sudo apt install <deb name>
(roop) hobi@hobi:~/roop$ python run.py --execution-provider rocm --execution-threads
usage: run.py [-h] [-s SOURCE_PATHS] [-t TARGET_PATHS] [-o OUTPUT_PATH] [--frame-processor {face_swapper,face_enhancer} [{face_swapper,face_enhancer} ...]] [--keep-fps] [--keep-audio] [--keep-frames] [--keep-filenames] [--many-faces]
              [--video-encoder {libx264,libx265,libvpx-vp9}] [--video-quality [0-51]] [--max-memory MAX_MEMORY] [--execution-provider {rocm,cpu} [{rocm,cpu} ...]] [--execution-threads EXECUTION_THREADS] [-v]
run.py: error: argument --execution-threads: expected one argument
(roop) hobi@hobi:~/roop$ python run.py --execution-provider rocm --execution-threads 2
[ROOP.CORE] Creating temp resources...
[ROOP.CORE] Extracting frames...
Applied providers: ['ROCMExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'ROCMExecutionProvider': {'tunable_op_tuning_enable': '0', 'do_copy_in_default_stream': '1', 'miopen_conv_exhaustive_search': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'gpu_external_empty_cache': '0', 'gpu_external_free': '0', 'miopen_conv_use_max_workspace': '1', 'gpu_mem_limit': '18446744073709551615', 'tunable_op_enable': '0', 'gpu_external_alloc': '0', 'device_id': '0'}}
find model: /home/hobi/.insightface/models/buffalo_l/1k3d68.onnx landmark_3d_68 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['ROCMExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'ROCMExecutionProvider': {'tunable_op_tuning_enable': '0', 'do_copy_in_default_stream': '1', 'miopen_conv_exhaustive_search': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'gpu_external_empty_cache': '0', 'gpu_external_free': '0', 'miopen_conv_use_max_workspace': '1', 'gpu_mem_limit': '18446744073709551615', 'tunable_op_enable': '0', 'gpu_external_alloc': '0', 'device_id': '0'}}
find model: /home/hobi/.insightface/models/buffalo_l/2d106det.onnx landmark_2d_106 ['None', 3, 192, 192] 0.0 1.0
Applied providers: ['ROCMExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'ROCMExecutionProvider': {'tunable_op_tuning_enable': '0', 'do_copy_in_default_stream': '1', 'miopen_conv_exhaustive_search': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'gpu_external_empty_cache': '0', 'gpu_external_free': '0', 'miopen_conv_use_max_workspace': '1', 'gpu_mem_limit': '18446744073709551615', 'tunable_op_enable': '0', 'gpu_external_alloc': '0', 'device_id': '0'}}
find model: /home/hobi/.insightface/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0
Applied providers: ['ROCMExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'ROCMExecutionProvider': {'tunable_op_tuning_enable': '0', 'do_copy_in_default_stream': '1', 'miopen_conv_exhaustive_search': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'gpu_external_empty_cache': '0', 'gpu_external_free': '0', 'miopen_conv_use_max_workspace': '1', 'gpu_mem_limit': '18446744073709551615', 'tunable_op_enable': '0', 'gpu_external_alloc': '0', 'device_id': '0'}}
find model: /home/hobi/.insightface/models/buffalo_l/genderage.onnx genderage ['None', 3, 96, 96] 0.0 1.0
Applied providers: ['ROCMExecutionProvider', 'CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}, 'ROCMExecutionProvider': {'tunable_op_tuning_enable': '0', 'do_copy_in_default_stream': '1', 'miopen_conv_exhaustive_search': '0', 'arena_extend_strategy': 'kNextPowerOfTwo', 'gpu_external_empty_cache': '0', 'gpu_external_free': '0', 'miopen_conv_use_max_workspace': '1', 'gpu_mem_limit': '18446744073709551615', 'tunable_op_enable': '0', 'gpu_external_alloc': '0', 'device_id': '0'}}
find model: /home/hobi/.insightface/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5
set det-size: (640, 640)

rocBLAS error: Cannot read /opt/rocm-5.4.0/lib/rocblas/library/TensileLibrary.dat: No such file or directory
Aborted (core dumped)

Environment

Hardware description
CPU Rzyen 5 5600
GPU Radeon 6700XT
Software version
rocm-core v5.4.0.50400-72~22.04
rocblas v2.46.0.50400-72~22.04

Make sure that ROCm is correctly installed and to capture detailed environment information run the following command:

printf '=== environment\n' > environment.txt &&
printf '\n\n=== date\n' >> environment.txt && date >> environment.txt &&
printf '\n\n=== Linux Kernel\n' >> environment.txt && uname -a  >> environment.txt &&
printf '\n\n=== rocm-smi' >> environment.txt && rocm-smi  >> environment.txt &&
printf '\n\n' >> environment.txt && hipconfig  >> environment.txt &&
printf '\n\n=== rocminfo\n' >> environment.txt && rocminfo  >> environment.txt &&
printf '\n\n=== lspci VGA\n' >> environment.txt && lspci | grep -i vga >> environment.txt

Getting this error: ```No LSB modules are available.



### Additional context
I am super new to machine learning and I am having a nightmare time of making things work with ROCm. Pretty much at the end of the rope here guys. Any help would be appreciated. Thank you.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 29 (7 by maintainers)

Most upvoted comments

Hi @slipperyslipped. Your GPU uses the gfx1031 instruction set, but the binaries distributed by AMD are not built for that architecture as it is not officially supported. However, the gfx1030 instruction set is identical to the gfx1031 instruction set in all but name. For this reason, there are ways to get the existing binaries running on your GPU.

As a workaround, I would recommend setting the environment variable export HSA_OVERRIDE_GFX_VERSION=10.3.0. This will cause your GPU to report that it supports the gfx1030 instruction set, which is included in the AMD-provided binaries. I’ve confirmed that this works correctly with rocBLAS on the RX 6750 XT. I believe this workaround is generally applicable to any discrete RDNA 2 GPUs.

@NaturalHate, build for gfx1030 and run with export HSA_OVERRIDE_GFX_VERSION=10.3.0 set in your environment.

Hi @slipperyslipped. Your GPU uses the gfx1031 instruction set, but the binaries distributed by AMD are not built for that architecture as it is not officially supported. However, the gfx1030 instruction set is identical to the gfx1031 instruction set in all but name. For this reason, there are ways to get the existing binaries running on your GPU.

As a workaround, I would recommend setting the environment variable export HSA_OVERRIDE_GFX_VERSION=10.3.0. This will cause your GPU to report that it supports the gfx1030 instruction set, which is included in the AMD-provided binaries. I’ve confirmed that this works correctly with rocBLAS on the RX 6750 XT. I believe this workaround is generally applicable to any discrete RDNA 2 GPUs.

how can i do this? where u put this command? thanks for the help…

Having an identical issue with my RX 5700XT rocBLAS error: Cannot read /home/extocine/sd-scripts/venv/lib/python3.10/site-packages/torch/lib/rocblas/library/TensileLibrary.dat: No such file or directory for GPU arch : gfx1010

FYI seeing what seems to be the same TensileLibrary.dat: Illegal seek issue on Ubuntu 22.04 LTS and Radeon Software for Linux 23.20.

GPU is a 7800 XT.

Stack from running a basic PyTorch example under GDB is shown below. I did have to override gfx version to either 11.0.0 or 11.0.1 for it to see GPU at all but I forget which.

rocBLAS error: Cannot read /home/redacted/venv/lib/python3.10/site-packages/torch/lib/rocblas/library/TensileLibrary.dat: Illegal seek

Thread 1 "python3" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352507392) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
(gdb) where
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352507392) at ./nptl/pthread_kill.c:44
ROCm/rocBLAS#1  __pthread_kill_internal (signo=6, threadid=140737352507392) at ./nptl/pthread_kill.c:78
ROCm/rocBLAS#2  __GI___pthread_kill (threadid=140737352507392, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
ROCm/rocBLAS#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
ROCm/rocBLAS#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
ROCm/rocBLAS#5  0x00007fff4e341ccf in rocblas_abort_once() () from /home/redacted/venv/lib/python3.10/site-packages/torch/lib/librocblas.so
ROCm/rocBLAS#6  0x00007fff4e341c49 in rocblas_abort () from /home/redacted/venv/lib/python3.10/site-packages/torch/lib/librocblas.so
ROCm/rocBLAS#7  0x00007fff4dc44633 in (anonymous namespace)::TensileHost::initialize(Tensile::hip::SolutionAdapter&, int) ()
   from /home/redacted/venv/lib/python3.10/site-packages/torch/lib/librocblas.so
ROCm/rocBLAS#8  0x00007fff4dc33929 in (anonymous namespace)::get_library_and_adapter(std::shared_ptr<Tensile::MasterSolutionLibrary<Tensile::ContractionProblem, Tensile::ContractionSolution> >*, std::shared_ptr<hipDeviceProp_t>*, int) () from /home/redacted/venv/lib/python3.10/site-packages/torch/lib/librocblas.so
ROCm/rocBLAS#9  0x00007fff4dc46b6c in rocblas_status_ runContractionProblem<float, float, float>(RocblasContractionProblem<float, float, float> const&, rocblas_gemm_algo_, int) ()
   from /home/redacted/venv/lib/python3.10/site-packages/torch/lib/librocblas.so
[SNIP]