tensorflow: Unable to Get Metric Result Using Mixed Precision and Mirrored Strategy
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- TensorFlow installed from (source or binary): Binary
- TensorFlow version (use command below): 2.6.2
- Python version: 3.7.12
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version: cuda_11.1.TC455_06.29190527_0
- GPU model and memory: NVIDIA RTX A6000 48GB
You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with:
- TF 1.0:
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" - TF 2.0:
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
Unable to call tf.keras.Metric.result() when using MixedPrecision.
We get the following error
NotImplementedError: in user code:
/home/zach/package/package_name/metrics.py:113 result *
true_mean = self._true_sum / self._count
/home/zach/conda/envs/gent/lib/python3.7/site-packages/keras/mixed_precision/autocast_variable.py:412 __truediv__
return self.read_value() / o
/home/zach/conda/envs/gent/lib/python3.7/site-packages/keras/mixed_precision/autocast_variable.py:113 read_value
val = self._variable.read_value()
/home/zach/conda/envs/gent/lib/python3.7/site-packages/tensorflow/python/distribute/values.py:1303 read_value
"call `variable.read_value()` inside variable_sync_on_read_context is"
NotImplementedError: call `variable.read_value()` inside variable_sync_on_read_context is not supported **Describe the
Expected behavior
Should be able to get metric when using MirroredStrategy and MixedPrecision
- Do you want to contribute a PR? (yes/no):
- Briefly describe your candidate solution(if contributing):
Standalone code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 17 (7 by maintainers)
The AutoCastVariable class overrides all the arithmetic ops and uses
read_value, which throws the error we see when it’s called in the wrong context. Using the TF ops will convert the inputs to tensors before doing the computation to avoid this. We should support using the usual python arithmetic ops though, and I’m guessing something wonky is happening with the various context managers