tensorflow: StringLookup layer does not retrieve vocabulary after saving and loading the model
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
2.11, 2.12, 2.13
Custom code
No
OS platform and distribution
Linux Ubuntu 20.04.1
Mobile device
No response
Python version
3.9.5
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
We noticed we could pickle the model right after building it, but unpickling would fail after saving and loading it from the disk. Upon further investigation, we realized the error was due to the vocabulary of the StringLookup layer, which was becoming an empty list after the tf.keras.models.load_model operation.
The unpickling issue happens on TF 2.11 onwards. The unpickling worked on TF 2.8, 2.9, and 2.10, even though the empty vocabulary issue was still there.
#--------------------
Using the minimal reproducible example below, before saving the model, if we inspect the StringLookup layer we get:
full_model.layers[1].get_config()
Out[9]: {'name': 'string_lookup',
'trainable': True,
'dtype': 'int64',
'invert': False,
'max_tokens': None,
'num_oov_indices': 1,
'oov_token': '[UNK]',
'mask_token': None,
'output_mode': 'int',
'sparse': False,
'pad_to_max_tokens': False,
'vocabulary': ListWrapper(['a', 'b']),
'idf_weights': None,
'encoding': 'utf-8'}
After saving and loading to the disk, we get:
full_model_loaded.layers[1].get_config()
Out[10]: {'name': 'string_lookup',
'trainable': True,
'dtype': 'int64',
'invert': False,
'max_tokens': None,
'num_oov_indices': 1,
'oov_token': '[UNK]',
'mask_token': None,
'output_mode': 'int',
'sparse': False,
'pad_to_max_tokens': False,
'vocabulary': ListWrapper([]),
'idf_weights': None,
'encoding': 'utf-8'}
#-----------------------
We were able to circumvent the issue by creating a new class as follows:
@tf.keras.utils.register_keras_serializable()
class MyStringLookup(tf.keras.layers.StringLookup):
def get_config(self):
base_config = super().get_config()
custom = {"vocabulary": self.get_vocabulary()}
return {**base_config, **custom}
However, it would be nice if we didn’t have to create this wrapper.
Standalone code to reproduce the issue
import tensorflow as tf
import pickle
model_input = tf.keras.Input(shape=(1,), dtype=tf.int64)
lookup = tf.keras.layers.StringLookup(vocabulary=['a', 'b'])(model_input)
output = tf.keras.layers.Dense(10)(lookup)
full_model = tf.keras.Model(model_input, output)
# this part should work
model_bytes = pickle.dumps(full_model)
model_recovered = pickle.loads(model_bytes)
# this part should throw an error
full_model.save("/tmp/temp_model")
full_model_loaded = tf.keras.models.load_model("/tmp/temp_model")
model_bytes_2 = pickle.dumps(full_model_loaded)
model_recovered_2 = pickle.loads(model_bytes_2)
Relevant log output
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
File <command-901450068412846>:1
----> 1 model_recovered_2 = pickle.loads(model_bytes_2)
File /databricks/python/lib/python3.9/site-packages/keras/saving/pickle_utils.py:48, in deserialize_model_from_bytecode(serialized_model)
46 model = saving_lib.load_model(filepath)
47 except Exception as e:
---> 48 raise e
49 else:
50 return model
File /databricks/python/lib/python3.9/site-packages/keras/saving/pickle_utils.py:46, in deserialize_model_from_bytecode(serialized_model)
40 f.write(serialized_model)
41 # When loading, direct import will work for most custom objects
42 # though it will require get_config() to be implemented.
43 # Some custom objects (e.g. an activation in a Dense layer,
44 # serialized as a string by Dense.get_config()) will require
45 # a custom_object_scope.
---> 46 model = saving_lib.load_model(filepath)
47 except Exception as e:
48 raise e
File /databricks/python/lib/python3.9/site-packages/keras/saving/experimental/saving_lib.py:196, in load_model(filepath, custom_objects)
194 h5_file.close()
195 except Exception as e:
--> 196 raise e
197 else:
198 return model
File /databricks/python/lib/python3.9/site-packages/keras/saving/experimental/saving_lib.py:183, in load_model(filepath, custom_objects)
181 config_dict = json.loads(config_json)
182 # Construct the model from the configuration file in the archive.
--> 183 model = deserialize_keras_object(config_dict, custom_objects)
184 h5_file = h5py.File(tf.io.gfile.join(temp_path, _VARS_FNAME), "r")
185 _print_h5_file(h5_file, action="loading")
File /databricks/python/lib/python3.9/site-packages/keras/saving/experimental/serialization_lib.py:318, in deserialize_keras_object(config, custom_objects)
315 # Instantiate the class from its config inside a custom object scope
316 # so that we can catch any custom objects that the config refers to.
317 with object_registration.custom_object_scope(custom_objects):
--> 318 return cls.from_config(inner_config)
File /databricks/python/lib/python3.9/site-packages/keras/engine/training.py:3114, in Model.from_config(cls, config, custom_objects)
3107 functional_model_keys = [
3108 "name",
3109 "layers",
3110 "input_layers",
3111 "output_layers",
3112 ]
3113 if all(key in config for key in functional_model_keys):
-> 3114 inputs, outputs, layers = functional.reconstruct_from_config(
3115 config, custom_objects
3116 )
3117 model = cls(
3118 inputs=inputs, outputs=outputs, name=config.get("name")
3119 )
3120 functional.connect_ancillary_layers(model, layers)
File /databricks/python/lib/python3.9/site-packages/keras/engine/functional.py:1470, in reconstruct_from_config(config, custom_objects, created_layers)
1468 # First, we create all layers and enqueue nodes to be processed
1469 for layer_data in config["layers"]:
-> 1470 process_layer(layer_data)
1471 # Then we process nodes in order of layer depth.
1472 # Nodes that cannot yet be processed (if the inbound node
1473 # does not yet exist) are re-enqueued, and the process
1474 # is repeated until all nodes are processed.
1475 while unprocessed_nodes:
File /databricks/python/lib/python3.9/site-packages/keras/engine/functional.py:1451, in reconstruct_from_config.<locals>.process_layer(layer_data)
1447 else:
1448 # Instantiate layer.
1449 from keras.layers import deserialize as deserialize_layer
-> 1451 layer = deserialize_layer(layer_data, custom_objects=custom_objects)
1452 created_layers[layer_name] = layer
1454 node_count_by_layer[layer] = int(_should_skip_first_node(layer))
File /databricks/python/lib/python3.9/site-packages/keras/layers/serialization.py:252, in deserialize(config, custom_objects)
215 """Instantiates a layer from a config dictionary.
216
217 Args:
(...)
249
250 """
251 populate_deserializable_objects()
--> 252 return serialization.deserialize_keras_object(
253 config,
254 module_objects=LOCAL.ALL_OBJECTS,
255 custom_objects=custom_objects,
256 printable_module_name="layer",
257 )
File /databricks/python/lib/python3.9/site-packages/keras/saving/legacy/serialization.py:527, in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
525 else:
526 with object_registration.CustomObjectScope(custom_objects):
--> 527 deserialized_obj = cls.from_config(cls_config)
528 else:
529 # Then `cls` may be a function returning a class.
530 # in this case by convention `config` holds
531 # the kwargs of the function.
532 custom_objects = custom_objects or {}
File /databricks/python/lib/python3.9/site-packages/keras/engine/base_layer.py:860, in Layer.from_config(cls, config)
844 @classmethod
845 def from_config(cls, config):
846 """Creates a layer from its config.
847
848 This method is the reverse of `get_config`,
(...)
858 A layer instance.
859 """
--> 860 return cls(**config)
File /databricks/python/lib/python3.9/site-packages/keras/layers/preprocessing/string_lookup.py:333, in StringLookup.__init__(self, max_tokens, num_oov_indices, mask_token, oov_token, vocabulary, idf_weights, encoding, invert, output_mode, sparse, pad_to_max_tokens, **kwargs)
329 del kwargs["dtype"]
331 self.encoding = encoding
--> 333 super().__init__(
334 max_tokens=max_tokens,
335 num_oov_indices=num_oov_indices,
336 mask_token=mask_token,
337 oov_token=oov_token,
338 vocabulary=vocabulary,
339 vocabulary_dtype=tf.string,
340 idf_weights=idf_weights,
341 invert=invert,
342 output_mode=output_mode,
343 sparse=sparse,
344 pad_to_max_tokens=pad_to_max_tokens,
345 **kwargs
346 )
347 base_preprocessing_layer.keras_kpl_gauge.get_cell("StringLookup").set(
348 True
349 )
File /databricks/python/lib/python3.9/site-packages/keras/layers/preprocessing/index_lookup.py:323, in IndexLookup.__init__(self, max_tokens, num_oov_indices, mask_token, oov_token, vocabulary_dtype, vocabulary, idf_weights, invert, output_mode, sparse, pad_to_max_tokens, **kwargs)
320 self.idf_weights_const = self.idf_weights.value()
322 if vocabulary is not None:
--> 323 self.set_vocabulary(vocabulary, idf_weights)
324 else:
325 # When restoring from a keras SavedModel, the loading code will
326 # expect to find and restore a lookup_table attribute on the layer.
327 # This table needs to be uninitialized as a StaticHashTable cannot
328 # be initialized twice.
329 self.lookup_table = self._uninitialized_lookup_table()
File /databricks/python/lib/python3.9/site-packages/keras/layers/preprocessing/index_lookup.py:510, in IndexLookup.set_vocabulary(self, vocabulary, idf_weights)
507 idf_weights = np.array(idf_weights)
509 if vocabulary.size == 0:
--> 510 raise ValueError(
511 f"Cannot set an empty vocabulary, you passed {vocabulary}."
512 )
514 oov_start = self._oov_start_index()
515 token_start = self._token_start_index()
ValueError: Cannot set an empty vocabulary, you passed [].
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 18 (2 by maintainers)
@brunolucatto ,
It seems a valid issue for me. It needs to dig more for the root cause.
Hi @brunolucatto ,
Please find the attached gist.
As you can see in the gist, when we serialize, the vocabulary values aren’t captured, it gives an empty list. Whereas the vocab size is getting captured. There is some issue in the source code. We’ll check and get back to you.
Thank you
I have done some work, on this bug. The issue seems to be in this file load.py.
As I have highlighted, parse from string is generating the data as nodes via proto buff.
The output nodes are as follows,
nodes { node_id: 1 node_path: “root.layer-0” identifier: “_tf_keras_input_layer” metadata: “{"class_name": "InputLayer", "name": "input_1", "dtype": "int64", "sparse": false, "ragged": false, "batch_input_shape": {"class_name": "tuple", "items": [null, 1]}, "config": {"batch_input_shape": {"class_name": "tuple", "items": [null, 1]}, "dtype": "int64", "sparse": false, "ragged": false, "name": "input_1"}}” version { producer: 2 min_consumer: 1 } } nodes { node_id: 2 node_path: “root.layer-1” identifier: “_tf_keras_layer” metadata: “{"name": "string_lookup", "trainable": true, "expects_training_arg": false, "dtype": "int64", "batch_input_shape": null, "stateful": false, "must_restore_from_config": true, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "StringLookup", "config": {"name": "string_lookup", "trainable": true, "dtype": "int64", "invert": false, "max_tokens": null, "num_oov_indices": 1, "oov_token": "[UNK]", "mask_token": null, "output_mode": "int", "sparse": false, "pad_to_max_tokens": false, "idf_weights": null, "vocabulary": null, "vocabulary_size": 3, "encoding": "utf-8", "has_input_vocabulary": true}, "inbound_nodes": [[["input_1", 0, 0, {}]]], "shared_object_id": 1, "build_input_shape": {"class_name": "TensorShape", "items": [null, 1]}}” version { producer: 2 min_consumer: 1 } } nodes { node_id: 3 node_path: “root.layer_with_weights-0” identifier: “_tf_keras_layer” metadata: “{"name": "dense", "trainable": true, "expects_training_arg": false, "dtype": "float32", "batch_input_shape": null, "stateful": false, "must_restore_from_config": false, "preserve_input_structure_in_config": false, "autocast": true, "class_name": "Dense", "config": {"name": "dense", "trainable": true, "dtype": "float32", "units": 10, "activation": "linear", "use_bias": true, "kernel_initializer": {"class_name": "GlorotUniform", "config": {"seed": null}, "shared_object_id": 2}, "bias_initializer": {"class_name": "Zeros", "config": {}, "shared_object_id": 3}, "kernel_regularizer": null, "bias_regularizer": null, "activity_regularizer": null, "kernel_constraint": null, "bias_constraint": null}, "inbound_nodes": [[["string_lookup", 0, 0, {}]]], "shared_object_id": 4, "input_spec": {"class_name": "InputSpec", "config": {"dtype": null, "shape": null, "ndim": null, "max_ndim": null, "min_ndim": 2, "axes": {"-1": 1}}, "shared_object_id": 7}, "build_input_shape": {"class_name": "TensorShape", "items": [null, 1]}}” version { producer: 2 min_consumer: 1 }
in node_id:2 the vocabulary is getting dropped where as it is present in file content. Issue seems to be with protobuff and parsing. It’s 2 Am and I have worked on this for straight 4 hours and am taking a break. If you can wait will resume working on it 🙂.