tensorflow: tf.keras computes incorrect loss values with 3+D data

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Yes. For a minimal example, run

from tensorflow import keras

layer = keras.layers.Input(shape=(1, 1, 1))
model = keras.models.Model(inputs=layer, outputs=layer)
model.compile(optimizer='adam', loss='poisson', metrics=['poisson'])
data = [[[[[1]]], [[[2]]], [[[3]]]]]
model.fit(x=data, y=data, batch_size=2, verbose=1, epochs=10)

and observe that loss and poisson values are different, and loss values vary:

Epoch 1/10
3/3 [==============================] - 1s 236ms/sample - loss: 0.6740 - poisson: 0.4393
Epoch 2/10
3/3 [==============================] - 0s 2ms/sample - loss: 0.6740 - poisson: 0.4393
Epoch 3/10
3/3 [==============================] - 0s 40ms/sample - loss: 0.5452 - poisson: 0.4393
Epoch 4/10
3/3 [==============================] - 0s 96ms/sample - loss: 0.5452 - poisson: 0.4393
Epoch 5/10
3/3 [==============================] - 0s 1ms/sample - loss: 0.9772 - poisson: 0.4393
Epoch 6/10
3/3 [==============================] - 0s 2ms/sample - loss: 0.6740 - poisson: 0.4393
Epoch 7/10
3/3 [==============================] - 1s 201ms/sample - loss: 0.6740 - poisson: 0.4393
Epoch 8/10
3/3 [==============================] - 0s 2ms/sample - loss: 0.6740 - poisson: 0.4393
Epoch 9/10
3/3 [==============================] - 0s 999us/sample - loss: 0.9772 - poisson: 0.4393
Epoch 10/10
3/3 [==============================] - 1s 327ms/sample - loss: 0.9772 - poisson: 0.4393

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
TensorFlow installed from (source or binary): pip install tensorflow
TensorFlow version (use command below): v1.13.0-rc1-19-gc865ec5621, 1.13.0-rc2
Python version: 3.7.2 x64
CUDA/cuDNN version: n/a
GPU model and memory: n/a

Describe the current behavior When fitting a model with loss="poisson", I would expect reported loss and poisson values to be identical.

Describe the expected behavior loss values are incorrect. They vary from epoch to epoch.

Code to reproduce the issue See above.

Other info / logs

More code examples and investigations at https://stackoverflow.com/q/54802328/880783

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (4 by maintainers)

Links to this issue

Why am I getting different values between loss functions and metrics in TensorFlow Keras? - Stack Overflow

Most upvoted comments

oO0oO0oO0o0o00 on Sep 21, 2020

same problem

HoltSpalding on Jun 16, 2021

@oO0oO0oO0o0o00 I have the same strange problem.

ofiryaish on Oct 9, 2020

I have a similar problem. I have written my own test_step(self, data):

    def test_step(self, data):
        # Unpack the data
        x, y, sample_weight = data_adapter.unpack_x_y_sample_weight(data)

        y_pred = self.student(x[0], training=False)
        student_loss = self.compute_loss(x[0], y, y_pred, sample_weight)
        distillation_loss = self.distillation_loss_fn(tf.nn.softmax(x[1] / self.temperature, axis=1), tf.nn.softmax(y_pred / self.temperature, axis=1))* self.temperature**2          
        loss = self.alpha *student_loss + (1 - self.alpha) *distillation_loss
        
        self.compute_metrics(x[0], y, y_pred, sample_weight)
       
        results = {m.name: m.result() for m in self.metrics}
        results.update(
            {"student_loss": student_loss, "distillation_loss": distillation_loss, "minimized_loss": loss}
        )
        return results

during the validation of my model in the fit call, I get these outputs for the validation:

Epoch 1: LearningRateScheduler setting learning rate to 0.004999999888241291.
Epoch 1/15
9050/9050 [==============================] - 31s 3ms/step - loss: 0.4114 - sparse_categorical_accuracy: 0.8437 - student_loss: 0.4114 - distillation_loss: 0.5873 - minimized_loss: 0.4378 - val_loss: 0.3046 - val_sparse_categorical_accuracy: 0.9034 - val_student_loss: 0.2388 - val_distillation_loss: 0.4188 - val_minimized_loss: 0.2658 - lr: 0.0050

Epoch 2: LearningRateScheduler setting learning rate to 0.004999999888241291.
Epoch 2/15
9050/9050 [==============================] - 30s 3ms/step - loss: 0.3719 - sparse_categorical_accuracy: 0.8597 - student_loss: 0.3720 - distillation_loss: 0.5534 - minimized_loss: 0.3992 - val_loss: 0.3187 - val_sparse_categorical_accuracy: 0.8923 - val_student_loss: 0.3967 - val_distillation_loss: 0.3904 - val_minimized_loss: 0.3958 - lr: 0.0050

I wonder why val_loss is not the same as val_student_loss, since they are supposed to be the same. When I call model.evaluate() after training, val_loss and val_student_loss are the same. I noticed that when the batch size for my generator is equal to the size of the dataset, the correct results are calculated. The generator is called in a tf.keras.utils.Sequence Object which is used for the validation data. I assume that the output is calculated based on the last batch only.

svh7rng on Mar 25, 2023