tensorflow: C API does not appear to register GPU XLA platform correctly

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on a mobile device: not mobile device.
  • TensorFlow installed from (source or binary): both
  • TensorFlow version (use command below): 2.4.1, 2.5.0, master branch(as of 6/24) —tested in standard 2.5.0 libtensorflow tarball as well as rebuilt source
  • Python version: C_API bug
  • Bazel version (if compiling from source): 3.7.2 == bazel version =============================================== Build label: 3.7.2- (@non-git) Build time: Mon May 24 14:27:25 2021 (1621866445) Build timestamp: 1621866445 Build timestamp as int: 1621866445
  • GCC/Compiler version (if compiling from source): gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
  • CUDA/cuDNN version: 11.0, libcudnn.so.8.0.5
  • GPU model and memory: Nvidia V100 16G

Describe the problem

The XLA_GPU ops do not appear to be registered correctly in any recent version of libtensorflow.so. This includes the standard 2.5.0 tarball, the tf-nightly as of this morning as well as my own compiles from 2.4.1 (patched) 2.5.0 and the master branch as of a day ago. The enclosed program demonstrates the problem. Running it with a saved-model in the current directory called “saved_model” results in a failure with the error messages enclosed below. This problem is not affected by any environment variable settings. setting “TF_XLA_FLAGS=–tf_xla_enable_xla_devices” will result in the XLA_CPU device being registered, but not the XLA_GPU device. In my environment all versions of tf-python register XLA ops correctly and do XLA just fine.


**Standalone code to reproduce the issue**
    ```c++
    #include "tensorflow/c/c_api.h"
    #include "tensorflow/c/c_api_experimental.h"
    #include <iostream>

    main()
    {
        auto sessionOpts = TF_NewSessionOptions();
        auto graph = TF_NewGraph();
        auto meta_graph_def = TF_NewBuffer();
        auto status = TF_NewStatus();
        const char *tags[] = {"serve"};

        TF_EnableXLACompilation(sessionOpts, 1u);   // All other methods of doing this also seem to fail

        auto session = TF_LoadSessionFromSavedModel(
            sessionOpts, /* RunOptions */ nullptr,
            "saved_model", tags,  1, graph, meta_graph_def, status);

        TF_DeleteSession(session, status);
        TF_DeleteSessionOptions(sessionOpts);
        TF_DeleteStatus(status);
        TF_DeleteBuffer(meta_graph_def);
        TF_DeleteGraph(graph);

        return 0;
    }
    ```

**Other info / logs** 

2021-06-25 06:40:17.652978: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-25 06:40:17.715316: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: saved_model
2021-06-25 06:40:18.241233: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2021-06-25 06:40:18.241281: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: saved_model
2021-06-25 06:40:18.241348: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-25 06:40:18.244220: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-25 06:40:18.245683: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-06-25 06:40:18.335973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0f:00.0 name: Tesla V100-DGXS-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2021-06-25 06:40:18.336004: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-25 06:40:18.339161: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-06-25 06:40:18.339198: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-06-25 06:40:18.341082: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-06-25 06:40:18.341400: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-06-25 06:40:18.343734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-06-25 06:40:18.344506: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-06-25 06:40:18.344662: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-06-25 06:40:18.349429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-25 06:40:18.349466: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-06-25 06:40:18.926592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-25 06:40:18.926630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2021-06-25 06:40:18.926642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2021-06-25 06:40:18.930171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14783 MB memory) -> physical GPU (device: 0, name: Tesla V100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 7.0)
2021-06-25 06:40:20.699787: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2021-06-25 06:40:20.971245: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2198735000 Hz
2021-06-25 06:40:24.264247: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at xla_ops.cc:406 : Not found: could not find registered platform with id: 0x7f9b890fdb28
2021-06-25 06:40:24.273317: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: fail: Not found: 2 root error(s) found.
  (0) Not found: could not find registered platform with id: 0x7f9b890fdb28
         [[{{node cluster_0_1/xla_compile}}]]
         [[cluster_0_1/data_as_ctrl/_2]]
  (1) Not found: could not find registered platform with id: 0x7f9b890fdb28
         [[{{node cluster_0_1/xla_compile}}]]
0 successful operations.
0 derived errors ignored.. Took 6558004 microseconds.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 4
  • Comments: 18 (4 by maintainers)

Most upvoted comments

Is there any movement on this? Is someone planning to fix this?

It cripples the C API for any model that requires XLA for speed, and has broken my project. Currently using @cbquillen’s workaround but that is not suitable for production.

@mihaimaruseac: Is there something concerned parties can do to help you get permission/bandwidth to work this issue, or give to another? This is a pretty egregious error.