tensorflow: Linking of rule '//tensorflow/contrib/factorization:gen_gen_clustering_ops_py_wrappers_cc' failed (missing -lcuda?)

Trying to build Tensorflow from a6f856b2f7920d4f74d7ca4e71967258423cc9f0 with CUDA and just started running into this issue:

____[7,182 / 7,674] Compiling tensorflow/core/graph/node_builder.cc
____[7,183 / 7,675] Compiling tensorflow/core/graph/costmodel.cc
____[7,245 / 7,715] Linking tensorflow/contrib/factorization/gen_gen_clustering_ops_py_wrappers_cc [for host]
____[7,246 / 7,715] Linking tensorflow/contrib/tensor_forest/hybrid/gen_training_ops_py_wrappers_cc [for host]
____[7,247 / 7,715] Linking tensorflow/cc/ops/lookup_ops_gen_cc [for host]
____[7,248 / 7,715] Linking tensorflow/contrib/tensor_forest/gen_gen_model_ops_py_py_wrappers_cc [for host]
____[7,249 / 7,715] Linking tensorflow/contrib/tensor_forest/gen_gen_tensor_forest_ops_py_wrappers_cc [for host]
____[7,250 / 7,715] Linking tensorflow/python/gen_set_ops_py_wrappers_cc [for host]
____[7,251 / 7,715] Linking tensorflow/python/gen_linalg_ops_py_wrappers_cc [for host]
ERROR: /build/tensorflow-git/src/tensorflow-cuda/tensorflow/contrib/factorization/BUILD:106:1: Linking of rule '//tensorflow/contrib/factorization:gen_gen_clustering_ops_py_wrappers_cc' failed (Exit 1).
/usr/bin/ld: warning: libcuda.so.1, needed by bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so, not found (try using -rpath or -rpath-link)
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemFree_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemsetD32Async'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuEventCreate'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuStreamAddCallback'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuModuleLoadFatBinary'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuCtxEnablePeerAccess'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemGetInfo_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuLaunchKernel'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuStreamSynchronize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuEventQuery'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuEventElapsedTime'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceCanAccessPeer'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuCtxSynchronize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceGetAttribute'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuFuncGetAttribute'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemcpyDtoH_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuStreamQuery'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDevicePrimaryCtxGetState'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuCtxSetCurrent'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuStreamWaitEvent'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuEventSynchronize'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuModuleUnload'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceGet'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemsetD32_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuCtxGetSharedMemConfig'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemFreeHost'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuFuncSetCacheConfig'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuStreamCreate'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemGetAddressRange_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuCtxGetDevice'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceGetProperties'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemcpyDtoHAsync_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceGetCount'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuModuleGetFunction'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemHostRegister_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDevicePrimaryCtxRelease'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemcpyHtoDAsync_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemcpyDtoD_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuModuleLoadDataEx'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDevicePrimaryCtxRetain'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemHostAlloc'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuInit'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDriverGetVersion'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceGetPCIBusId'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuEventRecord'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuPointerGetAttribute'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceTotalMem_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemsetD8_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceComputeCapability'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemAlloc_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDeviceGetName'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuDevicePrimaryCtxSetFlags'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemHostUnregister'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuModuleGetGlobal_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemcpyDtoDAsync_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuEventDestroy_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuOccupancyMaxActiveBlocksPerMultiprocessor'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuCtxGetCurrent'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuStreamDestroy_v2'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuCtxSetSharedMemConfig'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemsetD8Async'
bazel-out/host/bin/_solib_local/_U_S_Stensorflow_Scontrib_Sfactorization_Cgen_Ugen_Uclustering_Uops_Upy_Uwrappers_Ucc___Utensorflow/libtensorflow_framework.so: undefined reference to `cuMemcpyHtoD_v2'
collect2: error: ld returned 1 exit status
____Building complete.
____Elapsed time: 1247.002s, Critical Path: 405.23s

My previous successful build (I haven’t tried revisions in between) was from abfc9deb7. My build environment hasn’t changed significantly since then (no compiler, CUDA, or relevant library changes). I suspect it must be a recent change in the Tensorflow repo that’s the cause.

The only difference I see between abfc9deb7 and a6f856b2f7920d4f74d7ca4e71967258423cc9f0 in the tensorflow/contrib/factorization path is this:

diff --git a/tensorflow/contrib/factorization/kernels/BUILD b/tensorflow/contrib/factorization/kernels/BUILD
index 9a6d3c6f5..44eab5601 100644
--- a/tensorflow/contrib/factorization/kernels/BUILD
+++ b/tensorflow/contrib/factorization/kernels/BUILD
@@ -6,6 +6,8 @@ exports_files(["LICENSE"])

 package(default_visibility = ["//tensorflow:__subpackages__"])

+load("//tensorflow:tensorflow.bzl", "tf_cc_test")
+
 cc_library(
     name = "all_kernels",
     deps = [
@@ -50,7 +52,7 @@ cc_library(
     alwayslink = 1,
 )

-cc_test(
+tf_cc_test(
     name = "clustering_ops_test",
     srcs = ["clustering_ops_test.cc"],
     deps = [

I’m not super familiar with how Bazel works, but this looks innocuous to me. I guess something else must be implicated?

Any idea what could cause it to not add -lcuda on the link line? Or perhaps why it didn’t need to before but does now?

$ pacman -Q | grep -e gcc-mult -e ^cuda -e ^glibc -e ^cudnn -e ^gcc5 -e '^python ' -e bazel
bazel 0.5.4-1
cuda 8.0.61-3
cudnn6 6.0.21-2
gcc-multilib 7.2.1.20170910-1
gcc5 5.4.0-1
glibc 2.26-3
python 3.6.2-1