tensorflow: bug in rsqrt

---------------------------------------------------------------------------part 1----------------------------------------------------------------- my code:

import tensorflow as tf
import os
import numpy as np
np.set_printoptions(threshold=np.inf)

def test():
    with tf.device("/device:CPU:0"):
    # with tf.device('/gpu:1'):
        a=tf.constant([1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38])
        b = tf.rsqrt(a)
    return b

if __name__=='__main__':
    with tf.Session() as sess:
        print(sess.run(test()))

expect the right result: [9.223371843921341e+18, 9.223371843921341e+18, 9.223371843921341e+18, 9.223371843921341e+18, 9.223371843921341e+18, 9.223371843921341e+18, 9.223371843921341e+18, 9.223371843921341e+18, 9.223371843921341e+18]

in the env: tensorflow== 1.14.0 python=3.6.8 get the wrong result: [1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 9.223372e+18]

in the env: tensorflow== 1.14.0 python=3.6.8 get the wrong result: [1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 9.223372e+18]

in the env: tensorflow== 1.14.0 python=2.7.18 get the wrong result: [ inf inf inf inf inf inf inf inf 9.223372e+18]

note:if I use the gpu device, also get the wrong result ----------------------------------------------------------------------part 2----------------------------------------------------------------- my code2:

import tensorflow as tf
import os
import numpy as np
np.set_printoptions(threshold=np.inf)
import math

os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
def test():
    a=tf.constant([1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38,1.1754944e-38])
    b = tf.math.rsqrt(a)
    print(b)
    return
    
if __name__=='__main__':
    test()

in the env: tensorflow== 2.4.0 python=3.8.5 get the wrong result: [1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 1.383168e+19 9.223372e+18]

in the same env tensorflow== 2.4.0 python=3.8.5 if I use GPU, I mean os.environ[“CUDA_VISIBLE_DEVICES”] = “1” get the right result: [9.223372e+18 9.223372e+18 9.223372e+18 9.223372e+18 9.223372e+18 9.223372e+18 9.223372e+18 9.223372e+18 9.223372e+18]

----------------------------------------------------------------------part 3----------------------------------------------------------------- note: 1、the small shape tend to get the right result 2、similarity bug also in tf.floor

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (7 by maintainers)

Most upvoted comments

This is likely due to an Eigen approximation. If EIGEN_FAST_MATH = 1 (the default), then Eigen uses the fast reciprocal sqrt approximation, which is much faster, but less accurate (particularly as you deviate away from zero). See here.

If compiling TF from source, you can try explicitly disabling this (adding -DEIGEN_FAST_MATH=0 to the set of compile flags).

If not building TF from source, the work-around is to use tf.sqrt() instead, and do divisions as necessary.

cantonios on Sep 27, 2021