tensorflow: ValueError: Unable to save the object ListWrapper(...) (a list wrapper constructed to track trackable TensorFlow objects) when calling the method tf.keras.Model.save_weights

System information

Have I written custom code: yes
OS Platform and Distribution: macOS Catalina Version 10.15.3
TensorFlow installed from: binary (using pip in conda)
TensorFlow version (use command below): v2.1.0-rc2-17-ge5bf8de410 2.1.0
Python version: Python 3.6.10 :: Anaconda, Inc.

Note: probably related to this closed issue https://github.com/tensorflow/tensorflow/issues/35075

Describe the current behavior While saving a model tf.keras.Model with the function save_weights, the program raises the following error (full trace in the log section):

ValueError: Unable to save the object ListWrapper([ListWrapper([<tf.Variable ‘model_32/dense_4/kernel:0’ …)>, …]), ListWrapper([<tf.Variable ‘model_32/dense_6/kernel:0’ …)>, …])]) (a list wrapper constructed to track trackable TensorFlow objects). A list element was replaced (__setitem__, __setslice__), deleted (__delitem__, __delslice__), or moved (sort). In order to support restoration on object creation, tracking is exclusively for append-only data structures.

If you don’t need this list checkpointed, wrap it in a tf.contrib.checkpoint.NoDependency object; it will be automatically un-wrapped and subsequently ignored.

Note that the model can be saved before updating the model’s variables and the update works as expected. However once updated, if I try to save the model, it will crash.

Describe the expected behavior

The model should be saved without raising error
There shouldn’t be any reference to tf.contrib.checkpoint.NoDependency as contrib is not available in TF2.

Standalone code to reproduce the issue

import tensorflow as tf

class Model(tf.keras.Model):
    
    @property
    def groupedVariables(self):
        if self._var is None:
            self._var = []
            for denses in self._denses:
                self._var.append([])
                for d in denses:
                    self._var[-1] = self._var[-1] + d.trainable_variables                    
        return self._var
    ## ------------------------------------------------------------------------
    def __init__(self, ** kwargs):
        super(Model, self).__init__(** kwargs)

        self._optimizers = []
        self._denses     = []
        self._var        = None
        for copy in range(2):
            self._optimizers.append(tf.keras.optimizers.Adam())
            self._denses    .append([ tf.keras.layers.Dense(s) for s in [2,1]])
    ## ------------------------------------------------------------------------
    def call(self, x):
        y = []    
        for denses in self._denses:
            yy = x
            for d in denses: yy = d(yy)
            y.append(yy) 
        return y
    ## ------------------------------------------------------------------------
    def update(self, x, t):
        loss = []
        with tf.GradientTape() as tape:
            yy   = self(x)
            for y in zip(yy):
                y = y[0]
                l = tf.reduce_mean((t - y) ** 2)
                loss.append(l)

        var  = self.groupedVariables
        grad = tape.gradient(loss, var) 
        for g,v,o in zip(grad, var, self._optimizers):
            o.apply_gradients(zip(g,v))
## ----------------------------------------------------------------------------
m = Model()        
x = tf.zeros(shape = [1, 2], dtype = tf.float32)

print("Simple run then save ... ", end = "")
m(x)
m.save_weights("TMP/model") ## <== This works
print("done")

print("Update then save ....... ", end = "")
m.update(x, 0)
m.save_weights("TMP/model") ## <== This craches
print("done")

Other info / logs

The full error trace
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-62-6d7f6f1860e7> in <module>
     60 print("Update then save ....... ", end = "")
     61 m.update(x, 0)
---> 62 m.save_weights("TMP/model") ## <== This craches
     63 print("done")

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py in save_weights(self, filepath, overwrite, save_format)
   1121              'saved.\n\nConsider using a TensorFlow optimizer from `tf.train`.')
   1122             % (optimizer,))
-> 1123       self._trackable_saver.save(filepath, session=session)
   1124       # Record this checkpoint so it's visible from tf.train.latest_checkpoint.
   1125       checkpoint_management.update_checkpoint_state_internal(

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/util.py in save(self, file_prefix, checkpoint_number, session)
   1166     file_io.recursive_create_dir(os.path.dirname(file_prefix))
   1167     save_path, new_feed_additions = self._save_cached_when_graph_building(
-> 1168         file_prefix=file_prefix_tensor, object_graph_tensor=object_graph_tensor)
   1169     if new_feed_additions:
   1170       feed_dict.update(new_feed_additions)

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/util.py in _save_cached_when_graph_building(self, file_prefix, object_graph_tensor)
   1106     (named_saveable_objects, graph_proto,
   1107      feed_additions) = self._gather_saveables(
-> 1108          object_graph_tensor=object_graph_tensor)
   1109     if (self._last_save_object_graph != graph_proto
   1110         # When executing eagerly, we need to re-create SaveableObjects each time

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/util.py in _gather_saveables(self, object_graph_tensor)
   1074     """Wraps _serialize_object_graph to include the object graph proto."""
   1075     (named_saveable_objects, graph_proto,
-> 1076      feed_additions) = self._graph_view.serialize_object_graph()
   1077     if object_graph_tensor is None:
   1078       with ops.device("/cpu:0"):

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/graph_view.py in serialize_object_graph(self)
    377       ValueError: If there are invalid characters in an optimizer's slot names.
    378     """
--> 379     trackable_objects, path_to_root = self._breadth_first_traversal()
    380     return self._serialize_gathered_objects(
    381         trackable_objects, path_to_root)

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/graph_view.py in _breadth_first_traversal(self)
    197             % (current_trackable,))
    198       bfs_sorted.append(current_trackable)
--> 199       for name, dependency in self.list_dependencies(current_trackable):
    200         if dependency not in path_to_root:
    201           path_to_root[dependency] = (

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/graph_view.py in list_dependencies(self, obj)
    157     # pylint: disable=protected-access
    158     obj._maybe_initialize_trackable()
--> 159     return obj._checkpoint_dependencies
    160     # pylint: enable=protected-access
    161 

~usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/data_structures.py in _checkpoint_dependencies(self)
    507            "\n\nIf you don't need this list checkpointed, wrap it in a "
    508            "tf.contrib.checkpoint.NoDependency object; it will be "
--> 509            "automatically un-wrapped and subsequently ignored." % (self,)))
    510     if self._external_modification:
    511       raise ValueError(

ValueError: Unable to save the object ListWrapper([ListWrapper([<tf.Variable 'model_34/dense_12/kernel:0' shape=(2, 2) dtype=float32, numpy=
array([[ 0.4429444,  1.1044892],
       [ 0.515365 , -1.0957527]], dtype=float32)>, <tf.Variable 'model_34/dense_12/bias:0' shape=(2,) dtype=float32, numpy=array([0., 0.], dtype=float32)>, <tf.Variable 'model_34/dense_13/kernel:0' shape=(2, 1) dtype=float32, numpy=
array([[0.5423187 ],
       [0.27255356]], dtype=float32)>, <tf.Variable 'model_34/dense_13/bias:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>]), ListWrapper([<tf.Variable 'model_34/dense_14/kernel:0' shape=(2, 2) dtype=float32, numpy=
array([[-0.85178864,  1.0104183 ],
       [ 1.1649271 ,  1.1087018 ]], dtype=float32)>, <tf.Variable 'model_34/dense_14/bias:0' shape=(2,) dtype=float32, numpy=array([0., 0.], dtype=float32)>, <tf.Variable 'model_34/dense_15/kernel:0' shape=(2, 1) dtype=float32, numpy=
array([[-0.2449851 ],
       [ 0.84787834]], dtype=float32)>, <tf.Variable 'model_34/dense_15/bias:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>])]) (a list wrapper constructed to track trackable TensorFlow objects). A list element was replaced (__setitem__, __setslice__), deleted (__delitem__, __delslice__), or moved (sort). In order to support restoration on object creation, tracking is exclusively for append-only data structures.

If you don't need this list checkpointed, wrap it in a tf.contrib.checkpoint.NoDependency object; it will be automatically un-wrapped and subsequently ignored.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 2
Comments: 18 (6 by maintainers)

Commits related to this issue

Fixed name compatibility according to restoring model from checkpoint Changed dict keys from int to str according to this comment https://github.com/tensorflow/tensorflow/issues/36916#issuecomment-64... — committed to d61h6k4/automl by deleted user 4 years ago

Most upvoted comments

Hi,

I have been encountering the same issue, and found out by reading the source code for tracking data structures that the issue arises from Layer subclasses’ list attributes being automatically converted to ListWrapper at attribute setting time.

The solution, as documented per the source code, is to wrap non-trackable list attributes with the NonDependency class (which effectively results in an actual Python list being assigned as attribute instead of a ListWrapper object). That being said, this class is not part of the public TensorFlow API, so that using it requires importing from tensorflow.python, which is not something users are supposed to do (since this is not considered an open, stable API).

Another ugly solution is to force the assignment of a list attribute directly in the instance’s __dict__, instead of passing through its __setattr__ dunder method (which is where the conversion to a trackable ListWrapper happens). I.e. an alternative to doing self.my_list = NoDependency([]) is self.__dict__['my_list'] = [], but this feels like the kind of hack we should not want to promote.

I hope someone in the tensorflow team can either come up with a better solution than those above, or evaluate the pertinence of making (part of) the tracking data structures part of the public API, so that this issue may be solved.

+11

pandrey-fr on Mar 10, 2020

I have the same wrong error… I save the model with ‘h5’ no problem but save with ‘tf’ will cause this problem

chasers0-0 on Feb 20, 2020

@taki0112 As mentioned in my former reply, if you assign a list as attribute of a tf.keras.layers.Layer-inheriting instance, it will automatically be wrapped through a ListWrapper object, which as you notices causes some issues when said list does not comprise keras components. Note that when you want to store your model’s layers in a list attribute you do want it to be wrapped through this automatic behaviour!

I found two distinct workarounds for assigning a list without having it converted:

(1) Instead of assigning through dot syntax (self.my_list = []), directly manipulate the object’s __dict__, so that the conversion code is not triggered; i.e. use self.__dict__['my_list'] = [].

(2) Run from tensorflow.python.training.tracking.data_structures import NonDependency, then when assigning a list through dot syntax, wrap it with that wrapper first; i.e. use self.my_list = NonDependency(my_list).

Now, you should note that both these work-arounds are problematic. (1) has you access your object’s __dict__ directly, which in general is not something you want to do: it makes code harder to read, and it could have undesirable side effects, notably because you may be skipping desirable code from __setattr__. (2) has you import stuff from tensorflow.python which you should normally never do, since it is not part of the API and may change without warnings, making your code much harder to maintain.

Thus, I would actually be looking forward for someone in the TensorFlow dev team to give an eye on this issue, and either come out with a better solution or some future updates that allow users to have a workable and safe workaround.

pandrey-fr on May 18, 2020

@Nimoab @Saduf2019 You can not modify the tf.keras.layers.Layer list(self._var), only change to append the Layer list like the following pseudo-code: self._var.append(self._var[-1] + d.trainable_variables)

dengc367 on Mar 18, 2021

I get the same problem while i try to write a configurable UNet through parameters. So my up-sample is a list refer to the config file.

just simply change list to dictionary like:

self.rect_list = [ tf.keras.layers.Conv2D( filters = self.feature_model.outputs[i].shape[-1], kernel_size=(1, 1), activation=tf.nn.relu, name="{}_to_{}_1x1_conv".format(i+1,i), ) for i in range(f_layer_num-2,-1,-1)] to rect_list = [ tf.keras.layers.Conv2D( filters = self.feature_model.outputs[i].shape[-1], kernel_size=(1, 1), activation=tf.nn.relu, name="{}_to_{}_1x1_conv".format(i+1,i), ) for i in range(f_layer_num-2,-1,-1)] for i in range(len(rect_list)): self.rect_dirt['some_calling_name_format_like_l{}'.format(i)] = rect_list[i] and change calling like rect_list[i] to self.rect_dirt[‘some_calling_name_format_like_l{}’.format(i)] then it can be save

yomcoding on Jun 8, 2020