tensorflow: TF Hangs when calculating validation sample weights
Click to expand!
Issue Type
Performance
Have you reproduced the bug with TF nightly?
No
Source
binary
Tensorflow Version
unknown 2.9.2
Custom Code
Yes
OS Platform and Distribution
MacOS 12.4
Mobile device
NA
Python version
Python 3.10.5
Bazel version
NA
GCC/Compiler version
NA
CUDA/cuDNN version
NA
GPU model and memory
NA
Current Behaviour?
While using validation_data in model.fit, TF hangs when it’s finished the first epoch (but before updating the progress bar with the validation loss).
Specifically, this happens when a 3-tuple is passed to validation_data, where the third element is an array specifying the sample weights.
This happens even if the sample weights are all 1.0.
Standalone code to reproduce the issue
import numpy as np
import keras
import tensorflow as tf
import sklearn.model_selection
X = np.random.randint(low=300, high=900, size=(215699, 30, 30))
y = np.random.randint(low=0, high=51, size=(215699,))
X_trn, X_val, y_trn, y_val = sklearn.model_selection.train_test_split(X, y)
m = tf.keras.Sequential(
[
keras.layers.Input(shape=X.shape[1:]),
keras.layers.Flatten(),
keras.layers.Dense(units=64, activation="relu",),
keras.layers.Dense(units=64, activation="relu",),
keras.layers.Dense(len(np.unique(y))),
]
)
m.compile(
optimizer=keras.optimizers.Adam(1e-5),
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)
sample_weights_val = np.ones(y_val.shape)
history = m.fit(
X_trn,
y_trn,
validation_data=(X_val, y_val, sample_weights_val),
batch_size=128,
epochs=5,
)
Relevant log output
Epoch 1/5
2023-04-08 13:18:01.491389: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2023-04-08 13:18:01.621051: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
1264/1264 [==============================] - ETA: 0s - loss: 124.2101
At this point Tensorflow stops and I’ve got to interrupt it in order to get control back.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 15 (3 by maintainers)
Hi @beyarkay ,
If you want to use latest tensorflow-macos with metal plugin, you need to use legacy optimizer. The constraint you have mentioned for updating to new version is due to the optimizer issue.Suppose if you are using
keras.optimizers.Adam(1e-5)replace it withtf.keras.optimizers.legacy.Adam(1e-5)and it should work fine.Please try this and let us know of still having any problem. Thanks!