tf.extra_losses: undefined symbol: _ZNK10tensorflow8OpKernel11type_stringEv
I have Ubuntu 16.04, Python 3.5.2 in a virtual env, and Tensorflow 1.6 GPU. I have 2 1080ti. I have run the “cd build && cmake …” and the output is:
`-- The C compiler identification is GNU 5.4.0 – The CXX compiler identification is GNU 5.4.0 – Check for working C compiler: /usr/bin/cc – Check for working C compiler: /usr/bin/cc – works – Detecting C compiler ABI info – Detecting C compiler ABI info - done – Detecting C compile features – Detecting C compile features - done – Check for working CXX compiler: /usr/bin/c++ – Check for working CXX compiler: /usr/bin/c++ – works – Detecting CXX compiler ABI info – Detecting CXX compiler ABI info - done – Detecting CXX compile features – Detecting CXX compile features - done – Looking for pthread.h – Looking for pthread.h - found – Looking for pthread_create – Looking for pthread_create - not found – Looking for pthread_create in pthreads – Looking for pthread_create in pthreads - not found – Looking for pthread_create in pthread – Looking for pthread_create in pthread - found – Found Threads: TRUE
– Found CUDA: /usr/local/cuda-9.0 (found version “9.0”) – Found CWD: /home/vision/matthew/tf.extra_losses/build – Found GPU_CAPABILITY: gencode arch=compute_61,code=sm_61/home/vision/tf_16/lib/python3.5/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from
floattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters /home/vision/tf_16/lib/python3.5/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype fromfloattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters – Found TF_INC: /home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include – Found TF_INC_EXTERNAL: /home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/external/nsync/public – Found TF_LIB: /home/vision/tf_16/lib/python3.5/site-packages/tensorflow – Configuring done – Generating done – Build files have been written to: /home/vision/matthew/tf.extra_losses/build `
And Also ‘make’ command:
`[ 16%] Building NVCC (Device) object CMakeFiles/cuda_compile.dir/cuda_compile_generated_l_softmax_grad_op.cu.o /home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cu:181:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cu:181:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cu:181:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(57): warning: integer conversion resulted in a change of sign/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(304): warning: integer conversion resulted in a change of sign
/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(305): warning: integer conversion resulted in a change of sign
/home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cu:181:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(57): warning: integer conversion resulted in a change of sign/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(304): warning: integer conversion resulted in a change of sign
/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(305): warning: integer conversion resulted in a change of sign
/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/generated_message_reflection.h(685): warning: variable “unused” was set but never used
[ 33%] Building NVCC (Device) object CMakeFiles/cuda_compile.dir/cuda_compile_generated_l_softmax_op.cu.o /home/vision/matthew/tf.extra_losses/l_softmax_op.cu:131:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_op.cu:131:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_op.cu:131:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(57): warning: integer conversion resulted in a change of sign/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(304): warning: integer conversion resulted in a change of sign
/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(305): warning: integer conversion resulted in a change of sign
/home/vision/matthew/tf.extra_losses/common.h(32): warning: the “always_inline” attribute is ignored on non-inline functions
/home/vision/matthew/tf.extra_losses/common.h(46): warning: the “always_inline” attribute is ignored on non-inline functions
/home/vision/matthew/tf.extra_losses/common.h(52): warning: the “always_inline” attribute is ignored on non-inline functions
/home/vision/matthew/tf.extra_losses/l_softmax_op.cu:131:1: warning: multi-line comment [-Wcomment] // #define DEFINE_GPU_SPECS(T)
^ /home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(57): warning: integer conversion resulted in a change of sign/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(304): warning: integer conversion resulted in a change of sign
/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/arena_impl.h(305): warning: integer conversion resulted in a change of sign
/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/include/google/protobuf/generated_message_reflection.h(685): warning: variable “unused” was set but never used
/home/vision/matthew/tf.extra_losses/common.h(32): warning: the “always_inline” attribute is ignored on non-inline functions
/home/vision/matthew/tf.extra_losses/common.h(46): warning: the “always_inline” attribute is ignored on non-inline functions
/home/vision/matthew/tf.extra_losses/common.h(52): warning: the “always_inline” attribute is ignored on non-inline functions
Scanning dependencies of target extra_losses [ 50%] Building CXX object CMakeFiles/extra_losses.dir/l_softmax_op.cc.o /home/vision/matthew/tf.extra_losses/l_softmax_op.cc:274:0: warning: “REGISTER_CPU” redefined #define REGISTER_CPU(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_op.cc:268:0: note: this is the location of the previous definition #define REGISTER_CPU(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_op.cc:291:0: warning: “REGISTER_GPU” redefined #define REGISTER_GPU(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_op.cc:285:0: note: this is the location of the previous definition #define REGISTER_GPU(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_op.cc: In lambda function: /home/vision/matthew/tf.extra_losses/l_softmax_op.cc:62:40: warning: variable ‘num_dimensions’ set but not used [-Wunused-but-set-variable] shape_inference::DimensionHandle num_dimensions = c->Dim(features_shape, 1); ^ /home/vision/matthew/tf.extra_losses/l_softmax_op.cc: In lambda function: /home/vision/matthew/tf.extra_losses/l_softmax_op.cc:93:40: warning: variable ‘num_dimensions’ set but not used [-Wunused-but-set-variable] shape_inference::DimensionHandle num_dimensions = c->Dim(features_shape, 1); ^ [ 66%] Building CXX object CMakeFiles/extra_losses.dir/l_softmax_grad_op.cc.o /home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cc:293:0: warning: “REGISTER_CPU” redefined #define REGISTER_CPU(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cc:287:0: note: this is the location of the previous definition #define REGISTER_CPU(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cc:310:0: warning: “REGISTER_GPU” redefined #define REGISTER_GPU(T)
^ /home/vision/matthew/tf.extra_losses/l_softmax_grad_op.cc:304:0: note: this is the location of the previous definition #define REGISTER_GPU(T)
^ [ 83%] Building CXX object CMakeFiles/extra_losses.dir/common.cc.o In file included from /home/vision/matthew/tf.extra_losses/common.cc:22:0: /home/vision/matthew/tf.extra_losses/common.h:46:37: warning: always_inline function might not be inlinable [-Wattributes] attribute((always_inline)) bool is_pow2(const T x) ^ /home/vision/matthew/tf.extra_losses/common.h:32:39: warning: always_inline function might not be inlinable [-Wattributes] attribute((always_inline)) Target binary_cast(Source s) ^ /home/vision/matthew/tf.extra_losses/common.h:32:39: warning: always_inline function might not be inlinable [-Wattributes] /home/vision/matthew/tf.extra_losses/common.h:52:37: warning: always_inline function might not be inlinable [-Wattributes] attribute((always_inline)) bool is_aligned(const T ptr, const size_t alignment) ^ [100%] Linking CXX shared library libextra_losses.so [100%] Built target extra_losses `
but unfortunately got an error in test_op.py execution:
/home/vision/tf_16/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype fromfloattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Traceback (most recent call last): File "test_op.py", line 36, in <module> op_module = load_op_module(LIB_NAME) File "test_op.py", line 33, in load_op_module oplib = tf.load_op_library(lib_path) File "/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library lib_handle = py_tf.TF_LoadLibrary(library_filename, status) File "/home/vision/tf_16/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__ c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.NotFoundError: /home/vision/matthew/tf.extra_losses/build/libextra_losses.so: undefined symbol: _ZNK10tensorflow8OpKernel11type_stringEv
What should do?!
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (8 by maintainers)
@Alireza-Akhavan well… Thanks for your issue report.
@Alireza-Akhavan did you use cuda-9.0? where did you got ‘cuda_config.h’? please check that file, I have no problem for both following settings: 1.LargeMarginSoftmaxGradCudaKernel <<<config.block_count, config.thread_per_block, 0, d.stream()>>> 2. LargeMarginSoftmaxGradCudaKernel <<<5, 128, 0, d.stream()>>>
BTW, you should modify ‘LargeMarginSoftmaxGradCudaKernel’ not ‘LargeMarginSoftmaxCudaKernel’
@Alireza-Akhavan When you build from source(only support build using bazel not cmake), you should also add --cxxopt=“-D_GLIBCXX_USE_CXX11_ABI=0” if you build extra_losses by my cmake script. If you encounter ‘too many resources requested for launch’, that means there are limited registers to use, try to configure the block-size and thread_per_block in Kernel call according to your GPU card
@Alireza-Akhavan Maybe I have found out the problem. What’s your version of gcc? If you use higher version(>5) you should add extra option to use the older ABI. more here. Thanks again for your issue report, let me know if you still have other problems.
@Alireza-Akhavan It seems that it’s impossible that the linker failed to found type_string defination, there indeed exists defination here. Please let me know if you installed tensorflow from source. please make sure the file libtensorflow_framework.so exists under the foler ‘/home/vision/tf_16/lib/python3.5/site-packages/tensorflow’. Also make sure you run cmake with the activated virtual evironment. If the problem still exists please let me know, thank you.