tensorflow: AttributeError: '_TfDeviceCaptureOp' object has no attribute '_set_device_from_string'

Tensorflow Version: 1.14.0 python3.6 Num of GPU’s : 3

— Sample keras code to reproduce this error —

import tensorflow as tf
from keras.applications import Xception
from keras.utils import multi_gpu_model
import numpy as np

num_samples = 1000
height = 224
width = 224
num_classes = 1000

model = Xception(weights=None,
                     input_shape=(height, width, 3),
                     classes=num_classes)

gpus = tf.config.experimental.list_physical_devices('GPU')
print(gpus)
parallel_model = multi_gpu_model(model, gpus=2)

Error Message:


device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
device_string:  /device:GPU:0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/keras/utils/multi_gpu_utils.py", line 227, in multi_gpu_model
    outputs = model(inputs)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/network.py", line 564, in call
    output_tensors, _, _ = self.run_internal_graph(inputs, masks)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/network.py", line 721, in run_internal_graph
    layer.call(computed_tensor, **kwargs))
  File "/usr/local/lib/python3.6/dist-packages/keras/layers/normalization.py", line 185, in call
    epsilon=self.epsilon)
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 1858, in normalize_batch_in_training
    if not _has_nchw_support() and list(reduction_axes) == [0, 2, 3]:
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 291, in _has_nchw_support
    explicitly_on_cpu = _is_current_explicit_device('CPU')
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 266, in _is_current_explicit_device
    device = _get_current_tf_device()
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 247, in _get_current_tf_device
    g._apply_device_functions(op)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 4581, in _apply_device_functions
    op._set_device_from_string(device_string)
AttributeError: '_TfDeviceCaptureOp' object has no attribute '_set_device_from_string'
(arg: 2) 

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 4
  • Comments: 23 (3 by maintainers)

Commits related to this issue

Most upvoted comments

Solved, It’s a bug in tensorflow 1.14.0 When I downgraded to 1.13.1, it’s back to normal 😕 I haven’t figure out why 1.14.0 though. If someone solves this, it will help other idiots like me who after upgrading to TF 1.14 , unwittingly trained on a single gpu (instead of multi by default)

Downgrading to 1.13.1 worked for me as well!

Make sure when you downgrade, you downgrade other tensorflow packages that you have installed: if python 2:

pip install tensorflow==1.13.1
pip install tensorflow-gpu==1.13.1

If python 3:

pip3 install tensorflow==1.13.1
pip3 install tensorflow-gpu==1.13.1

I’m having the same issue, downgrading to 1.13.1 solved it for me, please fix.

With the release of keras-team/keras 2.3.0, this should be fixed in both keras and tf.keras.

@sephethus @ashkan-software This error is caused by keras, see the fix here