keras: NotFoundError and No algorithm worked

Using TensorFlow backend.
x_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
Train on 60000 samples, validate on 10000 samples
Epoch 1/12
2017-07-04 09:30:28.678755: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-04 09:30:28.678790: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-04 09:30:28.678801: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-04 09:30:28.678809: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-04 09:30:28.678817: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-07-04 09:30:28.950767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties: 
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:02:00.0
Total memory: 3.94GiB
Free memory: 3.79GiB
2017-07-04 09:30:29.160608: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x29a6ec0 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-07-04 09:30:29.161536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 1 with properties: 
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:03:00.0
Total memory: 3.94GiB
Free memory: 3.87GiB
2017-07-04 09:30:29.162085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0 1 
2017-07-04 09:30:29.162108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y Y 
2017-07-04 09:30:29.162116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 1:   Y Y 
2017-07-04 09:30:29.162130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:02:00.0)
2017-07-04 09:30:29.162143: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 980, pci bus id: 0000:03:00.0)
2017-07-04 09:30:32.258800: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
2017-07-04 09:30:33.259500: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
2017-07-04 09:30:34.259632: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
2017-07-04 09:30:35.259788: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
2017-07-04 09:30:36.259949: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
2017-07-04 09:30:37.260125: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
2017-07-04 09:30:38.260269: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
Traceback (most recent call last):
  File "mnist_cnn.py", line 67, in <module>
    validation_data=(x_test, y_test))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 870, in fit
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1507, in fit
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1156, in _fit_loop
    outs = f(ins_batch)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2269, in __call__
    **self.session_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: No algorithm worked!
	 [[Node: conv2d_2/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](conv2d_1/Relu, conv2d_2/kernel/read)]]
	 [[Node: mul_2/_39 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1029_mul_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'conv2d_2/convolution', defined at:
  File "mnist_cnn.py", line 51, in <module>
    model.add(Conv2D(64, (3, 3), activation='relu'))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 476, in add
    output_tensor = layer(self.outputs[0])
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 596, in __call__
    output = self.call(inputs, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/layers/convolutional.py", line 164, in call
    dilation_rate=self.dilation_rate)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 3138, in conv2d
    data_format='NHWC')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 670, in convolution
    op=op)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 338, in with_space_to_batch
    return op(input, num_spatial_dims, padding)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 662, in op
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 131, in _non_atrous_convolution
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 399, in conv2d
    data_format=data_format, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): No algorithm worked!
	 [[Node: conv2d_2/convolution = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](conv2d_1/Relu, conv2d_2/kernel/read)]]
	 [[Node: mul_2/_39 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1029_mul_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 15 (2 by maintainers)

Most upvoted comments

hello everybady, this can work well .

x_in = np.array([[ [[2], [1], [2], [0], [1]], [[1], [3], [2], [2], [3]], [[1], [1], [3], [3], [0]], [[2], [2], [0], [1], [1]], [[0], [0], [3], [1], [2]], ]]) kernel_in = np.array([ [ [[2, 0.1]], [[3, 0.2]] ], [ [[0, 0.3]],[[1, 0.4]] ], ]) x = tf.constant(x_in, dtype=tf.float32) kernel = tf.constant(kernel_in, dtype=tf.float32) tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='VALID') x_in.shape,kernel_in.shape

however ,wrong

x_in = np.array([[ [[0,1,0,1], [0,1,0,1], [0,1,0,1], [0,1,0,1]], [[0,1,0,1], [0,1,0,1], [0,1,0,1], [0,1,0,1]], [[0,1,0,1], [0,1,0,1], [0,1,0,1], [0,1,0,1]], ]]) x_in = tf.constant(x_in, dtype=tf.float32) kernel_in = np.array([ [ [[2, 0.1]], [[3, 0.2]] ], [ [[0, 0.3]],[[1, 0.4]] ], ]) kernel_in = tf.constant(kernel_in,dtype=tf.float32) x_in,kernel_in tf.nn.conv2d(input=x_in,filters = kernel_in,strides=[1,1,1,1],padding='SAME')

`--------------------------------------------------------------------------- NotFoundError Traceback (most recent call last) <ipython-input-26-31265a2d06bd> in <module> 19 kernel_in = tf.constant(kernel_in,dtype=tf.float32) 20 x_in,kernel_in —> 21 tf.nn.conv2d(input=x_in,filters = kernel_in,strides=[1,1,1,1],padding=‘SAME’)

~/anaconda3/envs/fish_detection_yoloV4/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py in conv2d_v2(input, filters, strides, padding, data_format, dilations, name) 1915 data_format=data_format, 1916 dilations=dilations, -> 1917 name=name) 1918 1919

~/anaconda3/envs/fish_detection_yoloV4/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py in conv2d(input, filter, strides, padding, use_cudnn_on_gpu, data_format, dilations, name, filters) 2012 data_format=data_format, 2013 dilations=dilations, -> 2014 name=name) 2015 2016

~/anaconda3/envs/fish_detection_yoloV4/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py in conv2d(input, filter, strides, padding, use_cudnn_on_gpu, explicit_paddings, data_format, dilations, name) 935 pass # Add nodes to the TensorFlow graph. 936 except _core._NotOkStatusException as e: –> 937 _ops.raise_from_not_ok_status(e, name) 938 # Add nodes to the TensorFlow graph. 939 if not isinstance(strides, (list, tuple)):

~/anaconda3/envs/fish_detection_yoloV4/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name) 6651 message = e.message + (" name: " + name if name is not None else “”) 6652 # pylint: disable=protected-access -> 6653 six.raise_from(core._status_to_exception(e.code, message), None) 6654 # pylint: enable=protected-access 6655

~/anaconda3/envs/fish_detection_yoloV4/lib/python3.6/site-packages/six.py in raise_from(value, from_value)

NotFoundError: No algorithm worked! [Op:Conv2D]`

we can find difference in code .

yes, It’s good, I make a wrong mattr…

+8
fucker007 on Jul 9, 2020

My problem was that I called the model with an input_shape of (?,28,28,1) and later called it with (?,28,28,3).

+6
philipperemy on Sep 8, 2020

I have an GeForce RTX 2070 device and my problem was resolved by upgrading tf-nightly: install --upgrade tf-nightly

+4
vladGriguta on Jan 22, 2021

A code from other issue helped me to find a way to limit tensorflow GPU memory usage and solved the issue, please see:

google-research/text-to-text-transfer-transformer#57 (comment)

the code is:

gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
  # Restrict TensorFlow to only allocate 4GB of memory on the first GPU
  try:
    tf.config.experimental.set_virtual_device_configuration(
        gpus[0],
        [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)])
    logical_gpus = tf.config.experimental.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Virtual devices must be set before GPUs have been initialized
    print(e)
+3
otaviomguerra on Feb 19, 2021

It seems to be CUDA problems.

2017-07-04 09:30:32.258800: E tensorflow/stream_executor/cuda/cuda_driver.cc:1073] failed to get elapsed time between events: CUDA_ERROR_NOT_READY
+1
taehoonlee on Jul 4, 2017
Read more comments on GitHub
← keras: Not able to resume training after loading model + weights
keras: Predict_generator problem - could not broadcast input array →