tensorflow: OP_REQUIRES failed : Not found: No algorithm worked!

System information

• OS Platform and Distribution: Linux Ubuntu 20.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
• TensorFlow installed from (source or binary): binary (pip install)
• TensorFlow version (use command below): v1.12.1-41444-g9396e98574 2.4.0-dev20200913
• Python version: 3.8.2
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version: 11.0 and 8.0
• GPU model and memory: GTX 2060

Describe the current behavior The script crashes at prediction.

Describe the expected behavior The script shall compute prediction

Standalone code to reproduce the issue

from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.models import Sequential
import numpy as np

model = Sequential([
    ResNet50(include_top=False, weights="imagenet", input_shape=(224, 224, 3), pooling="avg"),
    Dropout(0.5),
    Dense(64, activation='relu'),
])

X = np.zeros((1, 224, 224, 3))
print(model.predict(X))

Note that if adding the line tf.config.set_visible_devices([], 'GPU'), this problem disappear

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

2020-09-14 21:29:32.163353: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2020-09-14 21:29:32.931061: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-09-14 21:29:32.931534: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-09-14 21:29:32.953860: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:32.954284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:09:00.0 name: GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.68GHz coreCount: 30 deviceMemorySize: 5.79GiB deviceMemoryBandwidth: 312.97GiB/s
2020-09-14 21:29:32.954301: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2020-09-14 21:29:32.956135: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2020-09-14 21:29:32.957003: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-14 21:29:32.957172: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-14 21:29:32.959170: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-14 21:29:32.959632: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
2020-09-14 21:29:32.959728: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-09-14 21:29:32.959814: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:32.960277: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:32.960669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-14 21:29:32.960919: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-09-14 21:29:32.961335: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-09-14 21:29:32.961418: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:32.961827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:09:00.0 name: GeForce RTX 2060 computeCapability: 7.5
coreClock: 1.68GHz coreCount: 30 deviceMemorySize: 5.79GiB deviceMemoryBandwidth: 312.97GiB/s
2020-09-14 21:29:32.961841: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2020-09-14 21:29:32.961854: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2020-09-14 21:29:32.961864: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-09-14 21:29:32.961873: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-09-14 21:29:32.961882: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-09-14 21:29:32.961892: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.11
2020-09-14 21:29:32.961900: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-09-14 21:29:32.961949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:32.962375: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:32.962759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-09-14 21:29:32.962779: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
2020-09-14 21:29:33.321292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-14 21:29:33.321321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 
2020-09-14 21:29:33.321325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N 
2020-09-14 21:29:33.321485: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:33.321815: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:33.322113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-14 21:29:33.322399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5352 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2060, pci bus id: 0000:09:00.0, compute capability: 7.5)
2020-09-14 21:29:34.367791: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 1)
2020-09-14 21:29:34.384838: I tensorflow/core/platform/profile_utils/cpu_utils.cc:108] CPU Frequency: 3593095000 Hz
2020-09-14 21:29:34.803301: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.11
2020-09-14 21:29:35.075322: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-09-14 21:29:35.505290: W tensorflow/core/framework/op_kernel.cc:1774] OP_REQUIRES failed at conv_ops.cc:1114 : Not found: No algorithm worked!
Traceback (most recent call last):
  File "report_bug.py", line 13, in <module>
    print(model.predict(X))
  File "/opt/jupyterhub/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 102, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/opt/jupyterhub/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py", line 1584, in predict
    tmp_batch_outputs = self.predict_function(iterator)
  File "/opt/jupyterhub/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 787, in __call__
    result = self._call(*args, **kwds)
  File "/opt/jupyterhub/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py", line 853, in _call
    return self._concrete_stateful_fn._call_flat(
  File "/opt/jupyterhub/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/opt/jupyterhub/lib/python3.8/site-packages/tensorflow/python/eager/function.py", line 556, in call
    outputs = execute.execute(
  File "/opt/jupyterhub/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError:  No algorithm worked!
         [[node sequential/resnet50/conv1_conv/Conv2D (defined at report_bug.py:13) ]] [Op:__inference_predict_function_6649]

Function call stack:
predict_function