tensorflow: GPU MaxPool gradient ops do not yet have a deterministic XLA implementation
Issue type
Feature Request
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
binary
TensorFlow version
tf 2.16
Custom code
Yes
OS platform and distribution
Linux Ubuntu 22.04
Mobile device
No response
Python version
3.9.19
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
12.4/8.9.7.29
GPU model and memory
NVIDIA GeForce RTX 3090
Current behavior?
When TF deterministic was set, runtime exception was thrown at MaxPooling2D().
Standalone code to reproduce the issue
When TF deterministic was set, runtime exception was thrown at MaxPooling2D().
Relevant log output
Traceback (most recent call last):
File "/home/ws/miniconda3/envs/tf216/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3526, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-3dda39ff370e>", line 1, in <module>
runfile('/mnt/projects/Projects/Test_Classification/train_model.py', wdir='/mnt/projects/Projects/Test_Classification')
File "/opt/pycharm-community-2024.1/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/opt/pycharm-community-2024.1/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/mnt/projects/Projects/Test_Classification/train_model.py", line 956, in <module>
history = model.fit(x_train, y_train,
File "/home/ws/miniconda3/envs/tf216/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/ws/miniconda3/envs/tf216/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:
Detected at node gradient_tape/functional_1_1/max_pooling2d_4_1/MaxPool2d/MaxPoolGrad defined at (most recent call last):
<stack traces unavailable>
GPU MaxPool gradient ops do not yet have a deterministic XLA implementation.
[[{{node gradient_tape/functional_1_1/max_pooling2d_4_1/MaxPool2d/MaxPoolGrad}}]]
tf2xla conversion failed while converting __inference_one_step_on_data_13588[]. Run with TF_DUMP_GRAPH_PREFIX=/path/to/dump/dir and --vmodule=xla_compiler=2 to obtain a dump of the compiled functions.
[[StatefulPartitionedCall]] [Op:__inference_one_step_on_iterator_14045]
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 22 (3 by maintainers)
I wrote this guide when Keras had
jit_compile = None(Keras 2 or tf-keras) in the compile method. Now, Keras 3 usesjit_compile = "auto"(trying to use XLA) for the TensorFlow backend, which is the root of the error.You can simply pass
jit_compile = Falseif you’re using Keras 3.Same here. I guess, as enable_op_determinism() documentation states:
Can you please implement MaxPool2D deterministic?
AveragePooling2D layer does not have that problem. My Setting:
to make code deterministic, causes:
using that Tensorflow code:
tensorflow: 2.11.0 nvcc: NVIDIA ® Cuda compiler driver Copyright © 2005-2021 NVIDIA Corporation Built on Wed_Jun__9_20:32:38_PDT_2021 Cuda compilation tools, release 11.3, V11.3.122 Build cuda_11.3.r11.3/compiler.30059648_0
I have the same problem about MaxPooling2D. The following tutorial about reproducibility of keras confirmed that the same error occurs: https://keras.io/examples/keras_recipes/reproducibility_recipes/
I am having the same issue with MaxPool1D
Tested with Keras 3.0, Tensorflow 2.17. Still same error:
GPU MaxPool gradient ops do not yet have a deterministic XLA implementation. [[{{node gradient_tape/test-5321-Actor_1/enc_max_26_1/MaxPool2d/MaxPoolGrad}}]]For me MaxPool1D is not an option.
Hi. If possible, please also consider deterministic XLA for
MaxPool1D. Let me know in case a separate issue is needed here or in another repo.This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.