tensorflow: Tensorflow version 2.16.1 has retracing problem for keras.model.train_on_batch().
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
binary
TensorFlow version
2,16,1
Custom code
No
OS platform and distribution
windows 10
Mobile device
No response
Python version
3.11
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
The problem happens in both windows and WSL (Ubuntu )operating system. Tensorflow 2.15.1 and earlier versions don’t have the problem. The warning messages will be given after the train_on_batch() is called 5 and 6 times. The function is about 100 times slower in Tensorflow 2.16.1 in comparison to earlier versions.
Standalone code to reproduce the issue
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
import numpy as np
import tensorflow as tf
import tensorboard
import time
from tensorflow import keras
from datetime import datetime
from tensorflow.keras.utils import plot_model
from tensorflow.keras.layers import Dense
import random
class trn_sizes:
fm_length = 256
fm_step = 128
LSTMCell = 20
#features = 14
layerOne = 20
layerTwo = 16
layerThree = 8
cepsTone = 12
timeStep = 1 # is 1 only for now
batches = 250
batch_samples = batches*timeStep*fm_step
noise_insert_time = 2 # second
noise_chi2_df = 10
sampling_rate = 16000
time_per_batch = batches*timeStep*fm_step/sampling_rate # 2s for 250 batch size
noise_chi2_scale = noise_insert_time*sampling_rate/noise_chi2_df
trn_sizes = trn_sizes()
class inputBatch(object, ):
def __init__(self, trn_sizes):
self.state = 0 #
self.iirPinkSt = [0.0, 0.0, 0.0]
self.pinkCount = 0
self.pinkLeft = 0
self.pinkBlkCnt = 0
self.noisePower = 0.02
self.iirPinkB1 = [0.010000000000000, -0.018940429700000, 0.009585641562817, -0.000621320035262]
self.iirPinkA1 = [1.000000000000000, -2.479309080000000 , 1.985012853639686, -0.505600430025288]
self.remainingSamples = 0
self.labelIdx = 0 # index based on the batch
self.samplesToBeFill = trn_sizes.batch_samples
self.vadSampleIdx = 0 # index based on the audio file
self.vadLabel = np.zeros(trn_sizes.batches)
self.samplesOut = np.zeros(trn_sizes.batch_samples, dtype=float)
self.indexa = 0
def getBatch4(self, trn_sizes):
samples = np.random.normal(0, 0.1, (trn_sizes.batches-1)*trn_sizes.cepsTone)
mfccs = tf.convert_to_tensor(samples, dtype=tf.float32)
mfccs3 = tf.reshape(mfccs, [trn_sizes.batches-1, 1, trn_sizes.cepsTone], )
vadLabel3 = np.random.randint(2, size =(249, 1), dtype=np.int32)
vadLabelTf = tf.convert_to_tensor(vadLabel3, dtype=tf.float32)
return {mfccs3.ref(), vadLabelTf.ref()}
def create_model(trn_sizes):
inputs = keras.Input(shape=(1,trn_sizes.cepsTone), name="digits")
x0 = keras.layers.Dense(trn_sizes.layerOne, activation="relu")(inputs)
y1 = keras.layers.GRU(trn_sizes.LSTMCell)(x0)
x1 = keras.layers.Dense(trn_sizes.layerTwo, activation="relu")(y1)
x2 = keras.layers.Dense(trn_sizes.layerThree, activation="relu")(x1)
outputs = keras.layers.Dense(1, activation='sigmoid', name="predictions")(x2)
model = keras.Model(inputs=inputs, outputs=outputs)
return model
FLAGS = None
def main(FLAGS):
print('tensorflow version: ' + tf.__version__)
print(tf.version.GIT_VERSION, tf.version.VERSION)
dirName = os.path.dirname(os.path.abspath(__file__)) +'/MSLB0'
random.seed(11)
vad = inputBatch(trn_sizes)
vadModel = create_model(trn_sizes)
vadModel.compile(loss='binary_crossentropy',
optimizer=keras.optimizers.Adam(learning_rate=1e-4),
metrics=['accuracy'])
vadModel.build(input_shape = [None, 10, trn_sizes.cepsTone])
print('model length = ', len(vadModel.weights))
print(vadModel.summary())
batch_size=trn_sizes.batches;
tensorboard = keras.callbacks.TensorBoard(
#log_dir='/tmp/my_tf_logs',
log_dir= "logs\\scalars\\" + datetime.now().strftime("%Y%m%d-%H%M%S"),
histogram_freq=0,
write_graph=True,
update_freq="batch"
)
tensorboard.set_model(vadModel)
start_time1 = time.perf_counter()
for i in range(10):
mfccs3, vadLabel = vad.getBatch4(trn_sizes)
if i % 10 == 0:
print('mfccs3 value = ', tf.keras.backend.eval(mfccs3.deref()[1,:]) )
print('vadLabel value = ', tf.keras.backend.eval(vadLabel.deref()[0:3]) )
logs = vadModel.train_on_batch(mfccs3.deref(), vadLabel.deref())
if i %10 == 0:
print('string logs = ', str(logs))
end_time1 = time.perf_counter()
total_duration1 = end_time1 - start_time1
print( f"total_training time is { total_duration1: .2f}s ")
print("training is done!")
main(FLAGS)
Relevant log output
WARNING:tensorflow:5 out of the last 5 calls to <function TensorFlowTrainer.make_train_function.<locals>.one_step_on_iterator at 0x0000023C2F73E020> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 16 (4 by maintainers)
Commits related to this issue
- Now using tf-keras due to https://github.com/tensorflow/tensorflow/issues/67033 — committed to Tanfeil/FLY-SMOTE-CCMCB by Tanfeil 2 years ago
- Now using tf-keras due to https://github.com/tensorflow/tensorflow/issues/67033 — committed to Tanfeil/FLY-SMOTE-CCMCB by Tanfeil 2 years ago
- Now using tf-keras due to https://github.com/tensorflow/tensorflow/issues/67033 — committed to Tanfeil/FLY-SMOTE-CCMCB by Tanfeil 2 years ago
- Now using tf-keras due to https://github.com/tensorflow/tensorflow/issues/67033 — committed to Tanfeil/FLY-SMOTE-CCMCB by Tanfeil 2 years ago
- Now using tf-keras due to https://github.com/tensorflow/tensorflow/issues/67033 — committed to Tanfeil/FLY-SMOTE-CCMCB by Tanfeil 2 years ago
@KichangKim I have a temporally solution for this. It might affect Tensorflow’s eager/graph mode.
You can comment off line 142 and 143 of trainer.py in folder \site-packages\keras\src\backend\tensorflow.
@tilakrayal Tensorflow v2.17 has the same issue as v2.16.1. The problem is not fixed.