tensorflow: tensorflow.python.framework.errors_impl.InvalidArgumentError: 'func' argument to TF_GraphCopyFunction cannot be null

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 10.14.6
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v2.3.0-54-gfcc4b966f1 2.3.1
• Python version: 3.7.6

Describe the expected behavior This is probably a duplicate of https://github.com/tensorflow/tensorflow/issues/42239 (which I cannot reopen), but I was able to isolate it as standalone tf.test.TestCase.

The following test itself is successful, but after it completes, an error is logged. (If you run it in an IDE, the output is not attributes to the test case, but visible in the full log)

While examining the original problem in our full model, I added some logging and I think a function named “__inference__destroyer_15572” was involved - I guess this is _destroyer() created in _list_functions_for_serialization() of CapturableResource which is a superclass of StaticHashTable.

I couldn’t catch it with a python breakpoint, probably because it happens when objects are released. Given that memory management is probably involved, I suspect the python version is important (which would explain why the example in the other issue was not reproducible)

It also still happens with tf-nightly v1.12.1-44401-g11bbaed857 2.4.0-dev20201023.

It doesn’t appear with v2.1.0-rc2-17-ge5bf8de410 2.1.0

Standalone code to reproduce the issue

import tensorflow as tf
import os
import shutil


class MyLookup(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.table_init = tf.lookup.KeyValueTensorInitializer(
            key_dtype=tf.int64,
            keys=[0, 1, 2],
            value_dtype=tf.string,
            values=["A", "B", "C"],
            name="table_init")
        self.index_to_kw = tf.lookup.StaticHashTable(self.table_init, "?")

    def call(self, inputs, **kwargs):
        return self.index_to_kw.lookup(inputs)


class TestSaveProblem(tf.test.TestCase):

    def determine_and_clear_test_workdir(self):
        testname = self.id()
        result = os.path.abspath(os.path.join(
            os.path.dirname(__file__), "tmp_test_workdir", testname))
        shutil.rmtree(result, ignore_errors=True)
        return result

    def testSaveProblem(self):
        export_dir = self.determine_and_clear_test_workdir() + "/saved_model"

        exampledata = [1, 2]

        input = tf.keras.layers.Input(shape=1, dtype=tf.int64)
        output = MyLookup(name='result')(input)
        model = tf.keras.Model(inputs=[input], outputs=[output])

        # save and load
        model.save(export_dir, save_format='tf', include_optimizer=False)
        loaded_model = tf.saved_model.load(export_dir, [tf.saved_model.SERVING]).signatures[
            'serving_default']

        # test after saving and loading (works!)
        loaded_result = loaded_model(tf.constant(exampledata, dtype=tf.int64))['result']
        self.assertAllEqual([b"B", b"C"], loaded_result)

if __name__ == '__main__':
    tf.test.main()

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

(tf2.3.1) till@mac-brychcyt [/Users/till/test] 21:08% python test_saveproblem.py 
Running tests under Python 3.7.6: /Users/till/tf2.3.1/bin/python
[ RUN      ] TestSaveProblem.testSaveProblem
2020-10-28 21:08:59.759328: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-10-28 21:08:59.770427: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe5afe7a390 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-10-28 21:08:59.770444: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-10-28 21:08:59.868931: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
INFO:tensorflow:Assets written to: /Users/till/test/tmp_test_workdir/__main__.TestSaveProblem.testSaveProblem/saved_model/assets
I1028 21:09:00.045232 4522530240 builder_impl.py:775] Assets written to: /Users/till/test/tmp_test_workdir/__main__.TestSaveProblem.testSaveProblem/saved_model/assets
INFO:tensorflow:time(__main__.TestSaveProblem.testSaveProblem): 0.37s
I1028 21:09:00.122746 4522530240 test_util.py:1973] time(__main__.TestSaveProblem.testSaveProblem): 0.37s
[       OK ] TestSaveProblem.testSaveProblem
[ RUN      ] TestSaveProblem.test_session
[  SKIPPED ] TestSaveProblem.test_session
----------------------------------------------------------------------
Ran 2 tests in 0.371s

OK (skipped=1)
Exception ignored in: <function CapturableResourceDeleter.__del__ at 0x12f193170>
Traceback (most recent call last):
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py", line 202, in __del__
    self._destroy_resource()
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
    *args, **kwds))
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/saved_model/function_deserialization.py", line 237, in restored_function_body
    return _call_concrete_function(function, inputs)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/saved_model/function_deserialization.py", line 74, in _call_concrete_function
    result = function._call_flat(tensor_inputs, function._captured_inputs)  # pylint: disable=protected-access
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 106, in _call_flat
    cancellation_manager)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1938, in _call_flat
    flat_outputs = forward_function.call(ctx, args_with_tangents)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 579, in call
    executor_type=executor_type)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/ops/functional_ops.py", line 1192, in partitioned_call
    f.add_to_graph(graph)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 495, in add_to_graph
    g._add_function(self)
  File "/Users/till/tf2.3.1/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3345, in _add_function
    gradient)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 'func' argument to TF_GraphCopyFunction cannot be null