tensorflow: TF Hangs when calculating validation sample weights

Click to expand!

Issue Type

Performance

Have you reproduced the bug with TF nightly?

No

Source

binary

Tensorflow Version

unknown 2.9.2

Custom Code

Yes

OS Platform and Distribution

MacOS 12.4

Mobile device

NA

Python version

Python 3.10.5

Bazel version

NA

GCC/Compiler version

NA

CUDA/cuDNN version

NA

GPU model and memory

NA

Current Behaviour?

While using validation_data in model.fit, TF hangs when it’s finished the first epoch (but before updating the progress bar with the validation loss).

Specifically, this happens when a 3-tuple is passed to validation_data, where the third element is an array specifying the sample weights.

This happens even if the sample weights are all 1.0.

Standalone code to reproduce the issue

import numpy as np
import keras
import tensorflow as tf
import sklearn.model_selection

X = np.random.randint(low=300, high=900, size=(215699, 30, 30))
y = np.random.randint(low=0, high=51, size=(215699,))
X_trn, X_val, y_trn, y_val = sklearn.model_selection.train_test_split(X, y)

m = tf.keras.Sequential(
    [
        keras.layers.Input(shape=X.shape[1:]),
        keras.layers.Flatten(),
        keras.layers.Dense(units=64, activation="relu",),
        keras.layers.Dense(units=64, activation="relu",),
        keras.layers.Dense(len(np.unique(y))),
    ]
)
m.compile(
    optimizer=keras.optimizers.Adam(1e-5),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)

sample_weights_val = np.ones(y_val.shape)

history = m.fit(
    X_trn,
    y_trn,
    validation_data=(X_val, y_val, sample_weights_val),
    batch_size=128,
    epochs=5,
)

Relevant log output

Epoch 1/5

2023-04-08 13:18:01.491389: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2023-04-08 13:18:01.621051: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.

1264/1264 [==============================] - ETA: 0s - loss: 124.2101

At this point Tensorflow stops and I’ve got to interrupt it in order to get control back.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (3 by maintainers)

Most upvoted comments

Hi @beyarkay ,

If you want to use latest tensorflow-macos with metal plugin, you need to use legacy optimizer. The constraint you have mentioned for updating to new version is due to the optimizer issue.Suppose if you are using keras.optimizers.Adam(1e-5) replace it with tf.keras.optimizers.legacy.Adam(1e-5) and it should work fine.

Please try this and let us know of still having any problem. Thanks!