HIP: HIP separable compilation linking fails when passed archives

Found while working on implementing HIP support in CMake Tested with compilers provided with ROCm 3.5 and 3.7

Given the following CMake:

cmake_minimum_required(VERSION 3.18)

project(static-lib-rdc HIP)

set(CMAKE_HIP_FLAGS "-fgpu-rdc")
set(CMAKE_HIP_ARCHITECTURES gfx908)

add_library(static-lib file1.cxx file2.hip.cxx file3.hip.cpp file4.hip file5.cu)

add_executable(verify main.cu)
target_link_libraries(verify PUBLIC static-lib)
set_source_files_properties(
    file1.cxx file2.hip.cxx file3.hip.cpp file4.hip file5.cu main.cu
  PROPERTIES
    LANGUAGE "HIP"
)

We get the following error when trying to link the verify executable

/opt/rocm/bin/hipcc -fgpu-rdc --amdgpu-target=gfx908 CMakeFiles/verify.dir/main.cu.o -o verify  libstatic-lib.a 
clang-11: error: no such file or directory: '/tmp/hipcc0wDdnRUU/file1.cxx.o /tmp/hipcc0wDdnRUU/file2.hip.cxx.o /tmp/hipcc0wDdnRUU/file3.hip.cpp.o /tmp/hipcc0wDdnRUU/file4.hip.o /tmp/hipcc0wDdnRUU/file5.cu.o'

It looks something is going wrong and the clang-offload-bundler files are not being placed in the correct location or being removed before linking.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 21 (20 by maintainers)

Most upvoted comments

@pfultz2 As far as I am aware the clang++ provided by rocm is not equivalent to upstream clang++ as it contains modifications that have not yet been accepted upstream. Therefore the ROCM provided clang compiler must be identified with a unique id and version number since it diverges from upstream.

Well there is no hipclang driver, its just clang++(from rocm or upstream) with -x hip.

It would be great to have Rocm provide clang++ or some other alias inside the rocm/bin folder as currently the only easily discoverable compiler is hipcc.

For cmake, clang++ with the hip::device cmake target should be preferred over hipcc

Yes long term would be to have CMake understand the following:

upstream Clang aka COMPILER_ID Clang supports the HIP language ( -x hip )
rocm Clang aka COMPILER_ID HIPClang ( subject to change ) supports the HIP language

Due to how CMake’s internal compiler detection works, we simply can’t rely on the existence of hip::device target as compiler detection can occur before any find queries. The approach for upstream Clang support most likely will follow how CMake added support for Clang as a CUDA compiler in 3.18.

hipcc doesnt work with upstream static libs

Can you elaborate or point me towards more information on this?

robertmaynard on Aug 28, 2020

hipcc / hipclang as a unique compiler compared to upstream clang.

Well there is no hipclang driver, its just clang++(from rocm or upstream) with -x hip. For cmake, clang++ with the hip::device cmake target should be preferred over hipcc. hipcc is mainly for non-cmake users, and has lots of limitations(requires a lot custom env variables to find its dependencies, and doesnt work with upstream static libs).

pfultz2 on Aug 28, 2020

The clang++ compiler will accept these two flags. However just passing the path to static lib is not enough, I think you have to also have -l too.

upstream clang and hipclang both consider the line /opt/rocm/bin/hipcc -fgpu-rdc --amdgpu-target=gfx908 CMakeFiles/verify.dir/main.cu.o -o verify libstatic-lib.a to be entirely valid when gpu-rdc is disabled.

The clang++ compiler will accept these two flags. However just passing the path to static lib is not enough, I think you have to also have -l too.

Are you sure? We already are building with static libs in MIOpen, rocBLAS, etc using cmake, so the explicit path should be working?

Sorry, you’re both right, when gpu-rdc is disabled, giving the path to static lib works fine.

aaronenyeshi on Aug 28, 2020

The clang++ compiler will accept these two flags. However just passing the path to static lib is not enough, I think you have to also have -l too.

upstream clang and hipclang both consider the line /opt/rocm/bin/hipcc -fgpu-rdc --amdgpu-target=gfx908 CMakeFiles/verify.dir/main.cu.o -o verify libstatic-lib.a to be entirely valid when gpu-rdc is disabled.

robertmaynard on Aug 28, 2020

The original bug report is that the link lines CMake is currently generating for my work to get HIP language support currently don’t work with HIPClang. All build-systems are effect by this issue.

As a workaround, you may try this:

add_dependencies(verify static-lib)
target_link_libraries(verify PRIVATE -lstatic -L${CMAKE_CURRENT_BINARY_DIR})

I will discuss with my team about consuming the libstatic-lib.a as an input.

aaronenyeshi on Aug 28, 2020

It looks like switching the link line to the following does work:

hipcc -fgpu-rdc --amdgpu-target=gfx908 CMakeFiles/verify.dir/main.cu.o -o verify -lstatic-lib -L.

Unfortunately this isn’t a solution that is acceptable for CMake.

Link lines like these are ambiguous espescially when linking to multiple numerous libraries in different locations. Since we haven’t given explicit paths to any of the libraries we could errantly pick up the wrong static-lib when given multiple link directories. CMake transforms link lines to use explicit paths whenever to possible to avoid such ambiguities.

robertmaynard on Aug 28, 2020