tensorflow: Bug: Exception ignored in BaseSession.__del__

@ksrinivs64 try to delete session from keras backend:

from keras import backend as K

# ... code
K.clear_session()

Same bug with me, anyway, it didn’t happen every time, just occasionally.

it induced by different gc sequence, if python collect session first , the program will exit successfully, if python collect swig memory(tf_session) first, the program exit with failure.

you can force python to del session by:

del session

or if you are using keras, you cant get the session instance, you can run folloing code at end of your code: import gc; gc.collect()

So can someone tell me what the final fix is? python 3.5, keras with tf backend. same problem.

Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x7ffad5365eb8>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 696, in __del__
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/c_api_util.py", line 30, in __init__
TypeError: 'NoneType' object is not callable

Unfortunately, after b21c69a fix, this exception will come up oftenly (not every time.):

Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x7f78b6702a58>>
Traceback (most recent call last):
  File "/home/liusida/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 532, in __del__
AttributeError: 'NoneType' object has no attribute 'TF_DeleteStatus'

and sometimes:

Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x7f276ef81cf8>>
Traceback (most recent call last):
  File "/home/liusida/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 532, in __del__
UnboundLocalError: local variable 'status' referenced before assignment

Hello everyone,

Same issue still happening to me occasionally even though I cleared the TF session with K.clear_session() as @hosford42 suggested after the code was done. @Nimi42 also suggested that in #8652. I also tried clearing the session before each forward pass to no avail.

Adding import tensorflow as tf before importing Keras like @Ares513 suggested in #8652 didn’t work either.

I’m making inferences on multiple images, which are kicked off from separate bash scripts that are executing sequentially so I’m wondering if clearing the session would actually help since every forward pass of my net is invoked from a different process.

I’m using python 3.4, keras 2.1.6, and tensorflow 1.8.0 (no GPU support).

This is the error:

Exception ignored in: <bound method Session.__del__ of <tensorflow.python.client.session.Session object at 0x7f037640b198>> Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 707, in __del__ TypeError: 'NoneType' object is not callable

Don’t know if it may be related but this is also a warning I get each time I run my inference script:

/usr/lib/python3.4/importlib/_bootstrap.py:321: FutureWarning: Conversion of the second argument of issubdtype fromfloattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. return f(*args, **kwds)

#8652 is related and some also solved it by placing K.clear_session() after all the code was done. Note that contrary to what the author of #8652 states this issue is not closed and @naatje80 mentions it’s a Keras issue but this is a related Keras issue that was closed stating it’s a TF issue. This issue is also related and was also closed stating it’s a tensorflow issue.

This is a nagging issue which is very important for me to sort out due to an upcoming checkpoint for the xView dataset challenge (June 15). If anyone can shed some light on this I will greatly appreciate it.

Thanks in advance.

For those using tensorflow as keras backend, you may add the following function to your code. It is also useful in case you need to manage your memory.

import os
import importlib
from keras import backend as K

def set_keras_backend(backend):
    if K.backend() != backend:
        os.environ['KERAS_BACKEND'] = backend
        importlib.reload(K)
        assert K.backend() == backend
    if backend == "tensorflow":
        K.get_session().close()
        cfg = K.tf.ConfigProto()
        cfg.gpu_options.allow_growth = True
        K.set_session(K.tf.Session(config=cfg))
        K.clear_session()

set_keras_backend("tensorflow")  

Same issue happened occasionally.

Exception ignored in: <bound method BaseSession.del of <tensorflow.python.client.session.Session object at 0x7f88e5b52908>> Traceback (most recent call last): File “/home/wcai/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py”, line 701, in del TypeError: ‘NoneType’ object is not callable

Python 3.5.2 TensorFlow 1.3.0 GPU version installed by virtualenv Linux Mint 18 64-bit

Here’s code that reproduces the Keras problem for me from a recent build of my test system. Ubuntu 14.04, Python 3.4 (default OS version), TensorFlow 0.12.1, Keras 1.2.0. pima.csv is a copy of this file:

"""Verify that Keras works correctly.

Example comes from: http://machinelearningmastery.com/tutorial-first-neural-network-python-keras/
"""
from keras import models
from keras import layers
import numpy
import gc

# fix random seed for reproducibility
numpy.random.seed(7)

dataset = numpy.loadtxt('pima.csv', delimiter=',')
# First 8 columns are the input variables
X = dataset[:, 0:8]
# Last column is the output variable
Y = dataset[:, 8]

# Create fully-connected network with three layers.
# The last layer has one neuron to predict the class.
model = models.Sequential()
model.add(layers.Dense(12, input_dim=8, init='uniform', activation='relu'))
model.add(layers.Dense(8, init='uniform', activation='relu'))
model.add(layers.Dense(1, init='uniform', activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X, Y, nb_epoch=150, batch_size=10, verbose=False)

scores = model.evaluate(X, Y, verbose=False)
print('{}: {}%'.format(model.metrics_names[1], scores[1] * 100))

# Explicitly reap session to avoid an AttributeError sometimes thrown by
# TensorFlow on shutdown. See:
# https://github.com/tensorflow/tensorflow/issues/3388
gc.collect()

Update: Calling backend.clear_session() instead of gc.collect() does indeed seem to fix the problem. Yay for not relying on GC magic! But this still really ought to be fixed. 😃

Another update: ITOT the gc.collect() solution I attempted in my straight-up TensorFlow test ended up not working reliably either. So, I went from doing this:

import gc
with tf.Session() as session:
    session.run(...)
gc.collect()

To doing this:

with tf.Session() as session:
    session.run(...)
del session

It’s a nondeterminstic error, so we’ll see if that takes.

Another: TF1.0.0, Win10

Exception ignored in: <bound method BaseSession.del of <tensorflow.python.client.session.Session object at 0x000001D3B79EF358>> Traceback (most recent call last): File “lib\tensorflow_gpu\tensorflow\python\client\session.py”, line 582, in del UnboundLocalError: local variable ‘status’ referenced before assignment

@liusida Hi I have commented out try andfinally block of code:

try:
  status = tf_seesion.TF_NewStatus( )
  tf_session.TF_DeleteSession(self._session, status)
finally:
  tf_session_TF_DeleteStatus(status)

Hope it wouldn’t have drastic impact down the road.