tensorflow: Losses with Reduction.NONE do not keep the input shape (the result is averaged along the last axis)

@matgad @amitsrivastava78 The loss functions actually do not perform any reduction by default. The expectation is that the input passed to the functions will be atleast 2D. These functions return one loss value per sample as output.

Eg if y_true and y_pred have the shape (3, 3), return value will be of shape (3,). => we get one loss value per sample if y_true and y_pred have the shape (3, 2, 4), return value will be of shape (3, 2) => we get one loss value per sample, per timestep etc.

Sorry that the functions do not have any documentation about this currently. We are working on adding that, please feel free to contribute to the documentation if you are interested.

I just replied on the PR with this as well.

Why would issue be closed before updating the documentation? I just ran into the problem because documentation specifically said reduction=Reduction.NONE would give me same shape as I send in.

Hi @amitsrivastava78, thanks for the answer. The behavior that you described is the default one. But when you define a loss with the optional parameter reduction=tf.losses.Reduction.NONE, the last reduce_mean (or any other reduce) should be skipped. This issue only applies with None reduction. BinaryCrossentropy(reduction=tf.losses.Reduction.NONE) is just an example, the same behavior should be extended for any loss function.

@jvishnuvardhan I see that but it still doesn’t make sense to me. I discovered the tf.nn.sigmoid_cross_entropy by accident and it behaves exactly as one would expect when there is no reduction (i.e. element-wise loss scalars). @pavithrasv explanation doesn’t makes sense to me and the documentation doesn’t either. It’s is weird to me that no reduction actually leads to a loss in tensor dimension…I just cannot wrap my head around on what “sample” means in @pavithrasv explanation or the official documentation.

If I am comparing two 3x3 tensors with no reduction I have no idea why I am not getting a 3x3 loss tensor out. I really may be just misunderstanding what “sample” means here. It almost seems like by sample you mean row in y_pred and y_true tensors. Its as if you actually do compute an element-wise loss and once this is done you take the mean along the -1 dimension. But the fact that a mean (or sum) is taken means there is a reduction by definition. 😦

I really dislike this inflexibility. It would be nice if reduction=‘none’ actually meant no reduction in tensor dimension. This behavior means of all the useful losses that tf.losses already provides would need to be re-implemented or do the adding an extra dimension workaround that @matgad pointed out. Again reduction equals none should not mean the software takes an average for me. I’ve updated and confirmed my example above:

a = np.array([[.3, .4], [.5, .6]])
b = np.array([[.312, .433], [.555, .66]])

print('incorrect output shape:', tf.keras.losses.BinaryCrossentropy(reduction=tf.losses.Reduction.NONE,
                                                                    from_logits=True)(a, b))
print('+'*10)
print('correct output shape:', tf.nn.sigmoid_cross_entropy_with_logits(a, b))
v = tf.nn.sigmoid_cross_entropy_with_logits(a, b).numpy()
print(f"confirming naughty behavior of reduction='none':{v.mean(axis=-1)} ")

incorrect output shape: tf.Tensor([0.76368433 0.70590138], shape=(2,), dtype=float64)
++++++++++
correct output shape: tf.Tensor(
[[0.76766615 0.75970248]
 [0.73116606 0.6806367 ]], shape=(2, 2), dtype=float64)
++++++++++
confirming naughty behavior of reduction='none':[0.76368431 0.70590138] 

No reduction should mean no reduction. Even if the documentation points out the behavior it is still incredibly inflexible and frustrating.

@matgad , i have raised the PR and included your cases as unit test cases as well, Lets wait for this PR to merge, in the mean time you can use this PR in case you are blocked.

#27784

Regards Amit

@matgad ,i see your point, i think this is a BUG, irrespective of the reduction, the tensorflow is applying the reduce_mean to everything, in the coming days i will try to raise one PR to fix this issue.

Regards Amit