keras: Tensorflow backend - bug in model._make_predict_function(...)

I had this problem when doing inference in a different thread than where I loaded my model. Here’s how I fixed the problem:

Right after loading or constructing your model, save the TensorFlow graph:

graph = tf.get_default_graph()

In the other thread (or perhaps in an asynchronous event handler), do:

global graph
with graph.as_default():
    (... do inference here ...)

I learned about this from https://www.tensorflow.org/versions/r0.11/api_docs/python/framework.html#get_default_graph

avital’s solution works!

keras with tensorflow backend.

Details: global thread:

        self.model = load_model(model_path)
        self.model._make_predict_function()
        self.graph = tf.get_default_graph()

another thread:

        with self.graph.as_default():
            labels = self.model.predict(data)

Same problem here. Seems to work fine normally. When deployed as a webservice using Flask, get this error.

@Froskekongen could you describe how you fixed this in more detail? I’m having an exactly the same error however in a different program.

It seems to work when I do it manually in a REPL, however when I deploy it as a webservice it breaks.

@avital Thanks! It works well with tensorflow, but my model was built on keras. How to fix this problem on keras?

If you’re using Flask, setting DEBUG=Falsein my Flask app settings solved this problem for me.

It appears this bug had nothing to do with either keras or tensorflow, but rather how async events were handled by the webserver I am using.

Running the flask app with debug=False and threaded=False resolved the issue for me.

I think the proper way to handle this issue when loading multiple models is by manually specifying different graph and session for each loaded model. The code should be like this

graph = tf.Graph()
with graph.as_default():
       session = tf.Session()
       with session.as_default():
          ## your load model code
          ## for instance
          model = load_model('model.h5')

And you need to save each graph and session value to some variable for later prediction use.

graph_var = graph
session_var = session

and you can use above variables for prediction mode

with graph_var.as_default():
     with session_var.as_default():
            model.predict(INPUT)

Hopefully this can help.

参考这篇文章： https://zhuanlan.zhihu.com/p/27101000 就是说，当你引用模型后，随后进行一次预测，后面再用到时，就不会报错。

translate: refer to this article: https://zhuanlan.zhihu.com/p/27101000

To solve this problem, you have to predict once after load model ,like

import numpy as np
from keras.models import load_model

model = load_model('model_example.h5')
print('testing model:', model.predict(np.zeros((1, 299, 299, 3))))
...

to flesh out @rishabhbhardwaj answer, app.run(debug=False, threaded=False) fixes it for me

Probably noting that model._make_predict_function() will make Keras play nice with threads, and the additional with graph.as_default(): is only needed when you’re using anything with callbacks (by the look of it).

Shouldn’t this kind of stuff just be handled by keras though? Calling a private function is by definition a hack. It would be great if:

• model.compile() should also call model._make_predict_function() by default
• calls to model.predict(), model.train() do their own graph context handling.

class Serve(Base):

def start_socket_server(self, port, model_path):

    # Load model files
    self.model = load_model(model_path)
    self.model._make_predict_function()
    self.graph = tf.get_default_graph()

	...
	some code
	...

    while True:
        t = threading.Thread(target=self.client_thread, args=(conn, ))
        t.setDaemon(True)
        t.start()

def client_thread(self, conn):

	...
	some code
	...

    while True:
		...
		some code
		...
		# predict
		with self.graph.as_default():
			labels = self.model.predict(feature_segments_padded)

Running the webservice with gunicorn in sync mode solved the issue.

I had the same issue. Although my code was written in an asynchronous style, there was a single worker, so for practical purposes it was pretty synchronous. Calling model._make_predict_function() after load_model did the trick, but I also got the impression that load_model should have called this for me.

For me neither calling the private _make_predict_function() nor the withgraph.as_default() approach worked. Instead (as @j-o-d-o mentions) if I clear the session at the end of each async task, the problem is fixed.

from keras import backend as K
K.clear_session()

Reopening, since it actually seems to be a problem that should be handled.

the same problem and i solve it. thanks @avital my solution is https://justttry.github.io/justttry.github.io/not-an-element-of-Tensor-graph/

Same problem (model.predtict breaking) for me too, but it worked when i switched to theano backend from tensflow.

I also have the same error under the tensorflow backend, however, it works using the theano backend.
@jstypka @Froskekongen Have you found a solution to fix it?

Not sure if it’s relevant to the original question, but maybe it’ll be useful to others:

Based on this answer, the following resolved tf’s multithreading compatibility for me:

# on thread 1
session = tf.Session(graph=tf.Graph())
with session.graph.as_default():
    k.backend.set_session(session)
    model = k.models.load_model(filepath)

# on thread 2
with session.graph.as_default():
    k.backend.set_session(session)
    model.predict(x, **kwargs)

The novelty here is keeping both the Session and the Graph for other threads. The model is loaded in their “context” (instead of the default ones) and kept for other threads to use. (By default the model is loaded to the default Session and the default Graph) Another plus is that they’re kept in the same object - easier to handle.

For people who don’t want to import tf you can use directly the backend like:

graph = K.get_session().graph

The rest is the same.

@shafy Thanks, your solution of setting debug=False worked in my flask service. I didn’t need to use tf.get_default_graph or any other solution.

@shafy Thanks a lot! This is very useful information for everybody running keras on any Flask app…

I had this issue when running a model from a flask web application.

It turned out that I had configured it to run with gunicorn and gevent, something like this: gunicorn wsgi_search_service:app --bind 0.0.0.0:80 --config gunicorn_conf.py --workers 3 --worker-class gevent --reload

Removing --worker-class gevent from the above statement has resolved this problem.

I assume that gevent serializes data for its async operation - and as such, it has tried to serialize data related to the Keras model, which actually couldn’t be serialized.

After 3 hours of trying I also solved the issue thanks to @zhudaoruyi.

It turned out that this error occurs if the first call of model.predict is not on the same thread as the model instantiation.

Possible solution like @zhudaoruyi said is to call model.predict right after creating the model (on the same thread) or instantiate the model lazily right before the first prediction request (so the two calls will be on the same thread)

same here, same issue! Works fine in REPL, issues running it behind a webservice.

If anyone is still struggling with Keras, here is what I’ve done:

from keras import backend
with backend.get_session().graph.as_default() as g:
    model = load_model(MODEL_PATH)
    prediction = prediction_function(model, data)

For me the is only solvable if i load the model inside the flask @app.route POST method. Which means I reload the model on every request which is very inefficient. Loading the model as a global either in the beginning of the flask app script or as global in the main(), does not work.

Any ideas on how to solve this?

I’m still having this issue and the recommended fixes do not work for me. The only thing that works is loading the model and making a prediction with each prediction request. I’m requesting that this issue be reopened as no actual solution has been proposed, just several workarounds that seem to work intermittently.

When working with (async) Webservices and you get this error from time to time when loading and using a model on different requests, don’t forget K.clear_session()!

Amazing solution!! I also encounter this problem. In my way, I define the model in the model.py. On the other hand, I create the model instance in main.py Just do the revision in the main.py!!!

@SiddhardhaSaran hi dear. i know what you mean. but in my case, on uwsgi even i setting one process one thread it still no work, so i use gunicorn. anyway thanks your good advice.

@SiddhardhaSaran thanks for reply. but using lazy-apps=true will make your system overload, if your thread is more than one. now i use gunicorn, it work well to me.

Agreed, this is not really resolved. The current best solution is @dyngts which is basically “manage the backend’s sessions yourself”. My opinion is that keras should provide an API to predict and train one or more models without having to import tensorflow or theano separately to deal with their session management, thread locals or equivalent internals used for managing graphs.

@bdnettleton you might want to consider my approach here https://github.com/keras-team/keras/issues/2397#issuecomment-385317242 which provide more isolated graph leveraging session

This worked in the default flask development server, but it’s not working and the program is stuck/hangs at “predict” method in uwsgi + nginx with flask.

avital’s solution works!

keras with tensorflow backend.

Details: global thread:

        self.model = load_model(model_path)
        self.model._make_predict_function()
        self.graph = tf.get_default_graph()

another thread:

        with self.graph.as_default():
            labels = self.model.predict(data)

When I try this solution I got this Error InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor ‘Placeholder_366’ with dtype float and shape [2] [[Node: Placeholder_366 = Placeholderdtype=DT_FLOAT, shape=[2], _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]

In Flask I had to move my Rasa Agent setup code (which is using Keras with Tensorflow backend to predict) to the @app.before_first_request and then store the agent in the app variable to fix this issue for my project. Thank for everybody contributing above, it helped me a lot.

@shafy Disabling debug mode in flask works, thanks a lot!