keras: Models built with custom metrics can't be loaded

Models saved using custom metrics throw an exception on loading:

def x(y_true, y_pred):
    return 0.0 * y_pred

""" define model here """

model.compile(...,
              metrics=[x])
model.save('somefile')
break_here = load_model('somefile')

This produces:

Using Theano backend.
Traceback (most recent call last):
  File "test-case.py", line 14, in <module>
    break_here = load_model('somefile')
  File "/usr/local/lib/python3.5/dist-packages/keras/models.py", line 155, in load_model
    sample_weight_mode=sample_weight_mode)
  File "/usr/local/lib/python3.5/dist-packages/keras/models.py", line 547, in compile
    **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 662, in compile
    metric_fn = metrics_module.get(metric)
  File "/usr/local/lib/python3.5/dist-packages/keras/metrics.py", line 104, in get
    return get_from_module(identifier, globals(), 'metric')
  File "/usr/local/lib/python3.5/dist-packages/keras/utils/generic_utils.py", line 16, in get_from_module
    str(identifier))
Exception: Invalid metric: x

The only workaround seems to be to edit metrics.py directly, which is a little rough. Here’s a gist that reproduces the problem.

I’d like keras to ignore missing metrics when loading, instead of stopping. In an ideal world, it could even search the current scope for appropriate functions.

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 28
Comments: 17 (1 by maintainers)

Most upvoted comments

Doesn’t the custom_objects parameter for load_model solve most of these cases? In @leondz’s example:

load_model('somefile', custom_objects={'x': x})

+164

gvtulder on Jan 5, 2017

Am I missing something here?

I see the proposed solution as a workaround, but not a solution. I would rather like to load the model without the custom metrics - and knowlingly disregarding them.

Might this not be feasible in a scenario where I train a model and save it to solely use it for inference in a later stage? I wouldn’t need these custom metrics for inference, would I?

tsafs on Apr 22, 2020

@sebastianfast I second that - we should be able to load a model without custom metrics.

In TF 2, you can load the model without compilation. See https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model#arguments_1 .

This allows you to call .compile with a different set of metrics, such as standard metrics only, but it will lose your optimizer state.

Trezorro on Jun 18, 2020

Any update on this? Can the JSON workaround be used with model checkpoints?

CP121 on Dec 20, 2016