tensorflow: C API does not appear to register GPU XLA platform correctly
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on a mobile device: not mobile device.
- TensorFlow installed from (source or binary): both
- TensorFlow version (use command below): 2.4.1, 2.5.0, master branch(as of 6/24) —tested in standard 2.5.0 libtensorflow tarball as well as rebuilt source
- Python version: C_API bug
- Bazel version (if compiling from source): 3.7.2 == bazel version =============================================== Build label: 3.7.2- (@non-git) Build time: Mon May 24 14:27:25 2021 (1621866445) Build timestamp: 1621866445 Build timestamp as int: 1621866445
- GCC/Compiler version (if compiling from source): gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
- CUDA/cuDNN version: 11.0, libcudnn.so.8.0.5
- GPU model and memory: Nvidia V100 16G
Describe the problem
The XLA_GPU ops do not appear to be registered correctly in any recent version of libtensorflow.so. This includes the standard 2.5.0 tarball, the tf-nightly as of this morning as well as my own compiles from 2.4.1 (patched) 2.5.0 and the master branch as of a day ago. The enclosed program demonstrates the problem. Running it with a saved-model in the current directory called “saved_model” results in a failure with the error messages enclosed below. This problem is not affected by any environment variable settings. setting “TF_XLA_FLAGS=–tf_xla_enable_xla_devices” will result in the XLA_CPU device being registered, but not the XLA_GPU device. In my environment all versions of tf-python register XLA ops correctly and do XLA just fine.
**Standalone code to reproduce the issue**
```c++
#include "tensorflow/c/c_api.h"
#include "tensorflow/c/c_api_experimental.h"
#include <iostream>
main()
{
auto sessionOpts = TF_NewSessionOptions();
auto graph = TF_NewGraph();
auto meta_graph_def = TF_NewBuffer();
auto status = TF_NewStatus();
const char *tags[] = {"serve"};
TF_EnableXLACompilation(sessionOpts, 1u); // All other methods of doing this also seem to fail
auto session = TF_LoadSessionFromSavedModel(
sessionOpts, /* RunOptions */ nullptr,
"saved_model", tags, 1, graph, meta_graph_def, status);
TF_DeleteSession(session, status);
TF_DeleteSessionOptions(sessionOpts);
TF_DeleteStatus(status);
TF_DeleteBuffer(meta_graph_def);
TF_DeleteGraph(graph);
return 0;
}
```
**Other info / logs**
2021-06-25 06:40:17.652978: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-25 06:40:17.715316: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: saved_model
2021-06-25 06:40:18.241233: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-06-25 06:40:18.241281: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: saved_model
2021-06-25 06:40:18.241348: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-25 06:40:18.244220: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-25 06:40:18.245683: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-06-25 06:40:18.335973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0f:00.0 name: Tesla V100-DGXS-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-06-25 06:40:18.336004: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-25 06:40:18.339161: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-06-25 06:40:18.339198: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-06-25 06:40:18.341082: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-06-25 06:40:18.341400: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-06-25 06:40:18.343734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-06-25 06:40:18.344506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-06-25 06:40:18.344662: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-06-25 06:40:18.349429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-25 06:40:18.349466: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-25 06:40:18.926592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-25 06:40:18.926630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-06-25 06:40:18.926642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-06-25 06:40:18.930171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14783 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2021-06-25 06:40:20.699787: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-06-25 06:40:20.971245: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2198735000 Hz
2021-06-25 06:40:24.264247: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at xla_ops.cc:406 : Not found: could not find registered platform with id: 0x7f9b890fdb28
2021-06-25 06:40:24.273317: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: fail: Not found: 2 root error(s) found.
(0) Not found: could not find registered platform with id: 0x7f9b890fdb28
[[{{node cluster_0_1/xla_compile}}]]
[[cluster_0_1/data_as_ctrl/_2]]
(1) Not found: could not find registered platform with id: 0x7f9b890fdb28
[[{{node cluster_0_1/xla_compile}}]]
0 successful operations.
0 derived errors ignored.. Took 6558004 microseconds.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 4
- Comments: 18 (4 by maintainers)
Is there any movement on this? Is someone planning to fix this?
It cripples the C API for any model that requires XLA for speed, and has broken my project. Currently using @cbquillen’s workaround but that is not suitable for production.
@mihaimaruseac: Is there something concerned parties can do to help you get permission/bandwidth to work this issue, or give to another? This is a pretty egregious error.