TheRock: [Issue]: Linker Error: Undefined OpenMP Symbol `__kmpc_dispatch_deinit` When Building hipSPARSE on Windows
Problem Description
When building the hipSPARSE subproject on Windows as part of TheRock, specifically when linking hipsparse-bench.exe and hipsparse-test.exe, linker errors occur due to an undefined symbol: __kmpc_dispatch_deinit.
Error Log:
[1/8] Building sub-project hipSPARSE (in background)
FAILED: math-libs/BLAS/hipSPARSE/stamp/build.stamp E:/work/TheRock/build/math-libs/BLAS/hipSPARSE/stamp/build.stamp
C:\WINDOWS\system32\cmd.exe /C "cd /D E:\work\TheRock\build\math-libs\BLAS\hipSPARSE\build && E:\work\TheRock\.venv\Scripts\python.exe E:/work/TheRock/build_tools/teatime.py --log-timestamps --label hipSPARSE --interactive E:/work/TheRock/build/logs/hipSPARSE_build.log -- "C:/Program Files/CMake/bin/cmake.exe" -E env --unset=ROCM_PATH --unset=ROCM_DIR --unset=HIP_PATH --unset=HIP_DIR -- "C:/Program Files/CMake/bin/cmake.exe" --build E:/work/TheRock/build/math-libs/BLAS/hipSPARSE/build && "C:\Program Files\CMake\bin\cmake.exe" -E touch E:/work/TheRock/build/math-libs/BLAS/hipSPARSE/stamp/build.stamp"
[hipSPARSE] [0/2] Re-checking globbed directories...
[hipSPARSE] [1/3] Linking CXX executable clients\staging\hipsparse-bench.exe
[hipSPARSE] FAILED: clients/staging/hipsparse-bench.exe
[hipSPARSE] C:\WINDOWS\system32\cmd.exe /C "cd . && E:\work\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe -nostartfiles -nostdlib -DWIN32 -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -DNOMINMAX -fms-extensions -fms-compatibility -D_ENABLE_EXTENDED_ALIGNED_STORAGE -Wno-documentation-unknown-command -Wno-documentation-pedantic -Wno-unused-command-line-argument -Wno-ignored-attributes -Wno-unknown-attributes -Wno-duplicate-decl-specifier --hip-path=E:/work/TheRock/build/core/clr/dist --hip-device-lib-path=E:/work/TheRock/build/core/clr/dist/lib/llvm/amdgcn/bitcode -O3 -DNDEBUG -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -L E:/work/TheRock/build/third-party/sysdeps/windows/zlib/build/stage/lib/rocm_sysdeps/lib -L E:/work/TheRock/build/third-party/sysdeps/windows/zstd/build/stage/lib/rocm_sysdeps/lib -L E:/work/TheRock/build/compiler/amd-llvm/stage/lib/llvm/lib -L E:/work/TheRock/build/core/clr/stage/lib -Xlinker /subsystem:console -fuse-ld=lld-link clients/benchmarks/CMakeFiles/hipsparse-bench.dir/client.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_arguments_config.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_bench.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_bench_app.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_bench_cmdlines.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_routine.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/__/common/arg_check.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/__/common/unit.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/__/common/utility.cpp.obj clients/benchmarks/CMakeFiles/hipsparse-bench.dir/__/common/hipsparse_template_specialization.cpp.obj -o clients\staging\hipsparse-bench.exe -Xlinker /MANIFEST:EMBED -Xlinker /implib:clients\benchmarks\hipsparse-bench.lib -Xlinker /pdb:clients\staging\hipsparse-bench.pdb -Xlinker /version:0.0 library/hipsparse.lib -fopenmp=libomp E:/work/TheRock/build/core/clr/dist/lib/amdhip64.lib E:/work/TheRock/build/core/clr/dist/lib/llvm/lib/clang/19/lib/windows/clang_rt.builtins-x86_64.lib "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.41.34120/lib/x64/libomp.lib" -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -loldnames && cd ."
[hipSPARSE] lld-link: error: undefined symbol: __kmpc_dispatch_deinit
[hipSPARSE] >>> referenced by clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_routine.cpp.obj:(enum hipsparseStatus_t __cdecl testing_spmv_coo<int, float>(struct Arguments))
[hipSPARSE] >>> referenced by clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_routine.cpp.obj:(void __cdecl host_bsrmm<float>(int, int, int, int, enum hipsparseDirection_t, enum hipsparseOperation_t, enum hipsparseOperation_t, float, class std::vector<int, class std::allocator<int>> const &, class std::vector<int, class std::allocator<int>> const &, class std::vector<float, class std::allocator<float>> const &, class std::vector<float, class std::allocator<float>> const &, int, float, class std::vector<float, class std::allocator<float>> &, int, enum hipsparseIndexBase_t))
[hipSPARSE] >>> referenced by clients/benchmarks/CMakeFiles/hipsparse-bench.dir/hipsparse_routine.cpp.obj:(void __cdecl host_coomm<float, int>(int, int, int, int, enum hipsparseOperation_t, enum hipsparseOperation_t, float, int const *, int const *, float const *, float const *, int, enum hipsparseOrder_t, float, float *, int, enum hipsparseOrder_t, enum hipsparseIndexBase_t))
[hipSPARSE] >>> referenced 117 more times
[hipSPARSE] clang++: error: linker command failed with exit code 1 (use -v to see invocation)
[hipSPARSE] [2/3] Linking CXX executable clients\staging\hipsparse-test.exe
[hipSPARSE] FAILED: clients/staging/hipsparse-test.exe
[hipSPARSE] C:\WINDOWS\system32\cmd.exe /C "cd . && E:\work\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe -nostartfiles -nostdlib -DWIN32 -DWIN32_LEAN_AND_MEAN -D_CRT_SECURE_NO_WARNINGS -DNOMINMAX -fms-extensions -fms-compatibility -D_ENABLE_EXTENDED_ALIGNED_STORAGE -Wno-documentation-unknown-command -Wno-documentation-pedantic -Wno-unused-command-line-argument -Wno-ignored-attributes -Wno-unknown-attributes -Wno-duplicate-decl-specifier --hip-path=E:/work/TheRock/build/core/clr/dist --hip-device-lib-path=E:/work/TheRock/build/core/clr/dist/lib/llvm/amdgcn/bitcode -O3 -DNDEBUG -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -L E:/work/TheRock/build/third-party/sysdeps/windows/zlib/build/stage/lib/rocm_sysdeps/lib -L E:/work/TheRock/build/third-party/sysdeps/windows/zstd/build/stage/lib/rocm_sysdeps/lib -L E:/work/TheRock/build/compiler/amd-llvm/stage/lib/llvm/lib -L E:/work/TheRock/build/core/clr/stage/lib -Xlinker /subsystem:console -fuse-ld=lld-link @CMakeFiles\hipsparse-test.rsp -o clients\staging\hipsparse-test.exe -Xlinker /MANIFEST:EMBED -Xlinker /implib:clients\tests\hipsparse-test.lib -Xlinker /pdb:clients\staging\hipsparse-test.pdb -Xlinker /version:0.0 && C:\WINDOWS\system32\cmd.exe /C "cd /D E:\work\TheRock\build\math-libs\BLAS\hipSPARSE\build\clients\tests && "C:\Program Files\CMake\bin\cmake.exe" -E copy C:/Windows/System32/libomp140.x86_64.dll E:/work/TheRock/build/math-libs/BLAS/hipSPARSE/build/clients/staging/ && cd /D E:\work\TheRock\build\math-libs\BLAS\hipSPARSE\build\clients\tests && "C:\Program Files\CMake\bin\cmake.exe" -E copy C:/Windows/System32/libomp140d.x86_64.dll E:/work/TheRock/build/math-libs/BLAS/hipSPARSE/build/clients/staging/ && cd /D E:\work\TheRock\build\math-libs\BLAS\hipSPARSE\build\clients\tests && "C:\Program Files\CMake\bin\cmake.exe" -E copy E:/work/TheRock/math-libs/BLAS/hipSPARSE/rtest.py E:/work/TheRock/build/math-libs/BLAS/hipSPARSE/build/clients/staging/ && cd /D E:\work\TheRock\build\math-libs\BLAS\hipSPARSE\build\clients\tests && "C:\Program Files\CMake\bin\cmake.exe" -E copy E:/work/TheRock/math-libs/BLAS/hipSPARSE/rtest.xml E:/work/TheRock/build/math-libs/BLAS/hipSPARSE/build/clients/staging/""
[hipSPARSE] lld-link: error: undefined symbol: __kmpc_dispatch_deinit
[hipSPARSE] >>> referenced by clients/tests/CMakeFiles/hipsparse-test.dir/test_bsrmv.cpp.obj:(void __cdecl host_bsrmv<float>(enum hipsparseDirection_t, enum hipsparseOperation_t, int, int, int, float, int const *, int const *, float const *, int, float const *, float, float *, enum hipsparseIndexBase_t))
[hipSPARSE] >>> referenced by clients/tests/CMakeFiles/hipsparse-test.dir/test_bsrmv.cpp.obj:(void __cdecl host_bsrmv<double>(enum hipsparseDirection_t, enum hipsparseOperation_t, int, int, int, double, int const *, int const *, double const *, int, double const *, double, double *, enum hipsparseIndexBase_t))
[hipSPARSE] >>> referenced by clients/tests/CMakeFiles/hipsparse-test.dir/test_bsrmv.cpp.obj:(void __cdecl host_bsrmv<struct HIP_vector_type<float, 2>>(enum hipsparseDirection_t, enum hipsparseOperation_t, int, int, int, struct HIP_vector_type<float, 2>, int const *, int const *, struct HIP_vector_type<float, 2> const *, int, struct HIP_vector_type<float, 2> const *, struct HIP_vector_type<float, 2>, struct HIP_vector_type<float, 2> *, enum hipsparseIndexBase_t))
[hipSPARSE] >>> referenced 175 more times
[hipSPARSE] clang++: error: linker command failed with exit code 1 (use -v to see invocation)
[hipSPARSE] ninja: build stopped: subcommand failed.
Observations:
- The linker command for
hipsparse-bench.exeincludes both-fopenmp=libompand a direct path to MSVC’s OpenMP library:"C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.41.34120/lib/x64/libomp.lib". - The linker command for
hipsparse-test.exeinvolves a step to copyC:/Windows/System32/libomp140.x86_64.dllinto the staging directory. - A
dumpbin /exportsonC:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.41.34120/lib/x64/libomp.libandC:/Windows/System32/libomp140.x86_64.dllconfirmed that the symbol__kmpc_dispatch_deinitis not exported by these libraries. - The symbol
__kmpc_dispatch_deinitis defined (as an empty function) within TheRock’s source tree atcompiler/amd-llvm/openmp/runtime/src/kmp_dispatch.cpp.void __kmpc_dispatch_deinit(ident_t *loc, kmp_int32 gtid) {} - The build process for hipSPARSE appears to be using the OpenMP runtime from the Windows system, while the OpenMP runtime from amd-llvm does not seem to be built.
Operating System
Windows 11 Home [Version 10.0.26100.4061]
CPU
AMD Ryzen 5 5600X 6-Core Processor
GPU
AMD Radeon RX 7600 XT
ROCm Version
ROCm 6.5.25205-2cdd87330
ROCm Component
hipSPARSE
Steps to Reproduce
cmake -B build -GNinja -DTHEROCK_AMDGPU_FAMILIES=gfx110X-dgpu .
cmake --build build
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
No response
Additional Information
To avoid this issue, I specified -DBUILD_TESTING=OFF, but I encountered another problem, which has already been reported: #709
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 15 (6 by maintainers)
Note that amd-llvm has a substantial number of commits that are not upstream. I heard that https://github.com/ROCm/llvm-project/commit/dc2f7fea6de2a3fbb93704cf176d6dc9ef68a022 may be related here, but I saw the same build errors with it.
Some context on this from llvm: https://github.com/llvm/llvm-project/issues/98046 (This issue is for linux but it links to the fix for windows being to export the symbol) https://github.com/llvm/llvm-project/pull/97419
Seems the fix was put in July 2024.
I spent some time trying to build openmp on Windows but didn’t get far enough, so adding a minimum version check for now: https://github.com/ROCm/TheRock/pull/736
Correct, we do not currently build openmp on Windows as part of amd-llvm: https://github.com/ROCm/TheRock/blob/e9539fa10f44da1e5765090621ee17927b303c76/compiler/pre_hook_amd-llvm.cmake#L4-L21
We might be able to, but I see these errors after adding it to the list:
I just added some checks for MSVC in https://github.com/ROCm/TheRock/pull/719. Maybe we can query the version using variables like https://cmake.org/cmake/help/latest/variable/MSVC_VERSION.html.
Thank you for the detailed information. I’m going to reopen the issue until we update docs or checks for a recent enough version.
@stellaraccident Thank you. My Visual Studio is:
The latest version appears to be 17.14.2, so I will try upgrading.