tensorflow: NotImplementedError: Cannot convert a symbolic Tensor (args_0:0) to a numpy array.

System information

This is custom code
Running Google Colab on Mac
TensorFlow version 2.3.0
Python version 3.6.9
XLA_GPU hosted by Colab; memory_limit = 15695549568

Describe the current behavior I have a custom augmentation function written in Colab that worked normally until today. The last time I ran the entire code through was 09/08/2020 and the augmentation functioned performed the operations normally. Now, I receive an error that pertains to symbolic tensors, which I have never seen before.

Describe the expected behavior Augmentation function is meant to map over a batch of images and masks, taking in samples one at a time, converting them to NumPy arrays, performing some sort of augmentation, then returning them back as tensors.

Standalone code to reproduce the issue A link to the Colab notebook: https://colab.research.google.com/drive/1ujShezPjcveG2kg019c7zqweLtu1ESuW?usp=sharing A link to the training data I use in the notebook: https://drive.google.com/drive/folders/1ZS1wKjo692Lg7Vuhtr3akz0sYSOmZBKl

The particular section of code where the error stems from:

# Function used to perform "on-the-fly" augmentation during training.
# UPDATED ON 09/21/2020.

def augmentation(img, msk):

  # Call in skimage package, which will be used for transformations.
  from skimage.transform import rotate, AffineTransform, warp
  
  # Create some random floats, which will be used in augmentation steps.
  tilt = tf.random.uniform(shape = [], minval = -90, maxval = 90, dtype = tf.float32)
  dx = tf.random.uniform(shape = [], minval = -20, maxval = 20, dtype = tf.float32)
  dy = tf.random.uniform(shape = [], minval = -20, maxval = 20, dtype = tf.float32)
  
  # Cast image and mask to numpy arrays.
  img = np.array(img)
  msk = np.array(msk)

  # Use TensforFlow-style if conditionals, used to flip image and mask.
  img = tf.cond(tilt > 0, lambda: np.fliplr(img), lambda: np.flipud(img))
  msk = tf.cond(tilt > 0, lambda: np.fliplr(msk), lambda: np.flipud(msk))

  # Rotate the image and mask to some degree.
  img = rotate(img, angle = tilt, mode = 'reflect')
  msk = rotate(msk, angle = tilt, mode = 'reflect')

  # Write the conditions for an affine transformation.
  transform = AffineTransform(translation = (dx,dy))

  # Perform the affine transformation.
  img = warp(img, inverse_map = transform, mode = 'reflect')
  msk = warp(msk, inverse_map = transform, mode = 'reflect')
 
  # Convert the inputs back into tensors, put back into a tuple.
  finalTuple = (tf.convert_to_tensor(img), tf.convert_to_tensor(msk))

  return finalTuple

# Callback for data augmentation.
class aug(tf.keras.callbacks.Callback):
  def on_training_batch_begin(self, batch, logs = None):
    batch.map(augmentation, num_parallel_calls = 5)
    batch.shuffle(10)
    
# Callback for CSV logger (used for charting).
csv = tf.keras.callbacks.CSVLogger(f'/content/gdrive/My Drive/{today}_metrics.csv', separator=',', append=False)

# Callback for saving the model.
save_model_path = f'/content/gdrive/My Drive/{today}_wellpad_model_.h5'
cp = tf.keras.callbacks.ModelCheckpoint(filepath=save_model_path,
                                        monitor='val_loss',
                                        mode='min',
                                        save_best_only=True)

# Test the augmentation function on a batch of images.
test_batch = evaluation.take(1)

test_batch = test_batch.map(augmentation,num_parallel_calls=5) # ERROR OCCURS HERE.

# For plotting below.
test_batch = [(image,mask) for image, mask in test_batch]

plt.figure(figsize=(10,10))
plt.subplot(1,2,1)
# Show the original image.
plt.imshow(test_batch[0][0][0])
# Show the masked image.
plt.subplot(1,2,2)
plt.imshow(tf.squeeze(test_batch[0][1][0]))

Other info / logs Here is the full error I receive:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
<ipython-input-17-81c1086f76f0> in <module>()
      2 test_batch = evaluation.take(1)
      3 
----> 4 test_batch = test_batch.map(augmentation,num_parallel_calls=5)
      5 
      6 test_batch = [(image,mask) for image, mask in test_batch]

10 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in map(self, map_func, num_parallel_calls, deterministic)
   1700           num_parallel_calls,
   1701           deterministic,
-> 1702           preserve_cardinality=True)
   1703 
   1704   def flat_map(self, map_func):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, input_dataset, map_func, num_parallel_calls, deterministic, use_inter_op_parallelism, preserve_cardinality, use_legacy_function)
   4082         self._transformation_name(),
   4083         dataset=input_dataset,
-> 4084         use_legacy_function=use_legacy_function)
   4085     if deterministic is None:
   4086       self._deterministic = "default"

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in __init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
   3369       with tracking.resource_tracker_scope(resource_tracker):
   3370         # TODO(b/141462134): Switch to using garbage collection.
-> 3371         self._function = wrapper_fn.get_concrete_function()
   3372         if add_to_graph:
   3373           self._function.add_to_graph(ops.get_default_graph())

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in get_concrete_function(self, *args, **kwargs)
   2937     """
   2938     graph_function = self._get_concrete_function_garbage_collected(
-> 2939         *args, **kwargs)
   2940     graph_function._garbage_collector.release()  # pylint: disable=protected-access
   2941     return graph_function

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _get_concrete_function_garbage_collected(self, *args, **kwargs)
   2904       args, kwargs = None, None
   2905     with self._lock:
-> 2906       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
   2907       seen_names = set()
   2908       captured = object_identity.ObjectIdentitySet(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in wrapper_fn(*args)
   3362           attributes=defun_kwargs)
   3363       def wrapper_fn(*args):  # pylint: disable=missing-docstring
-> 3364         ret = _wrapper_helper(*args)
   3365         ret = structure.to_tensor_list(self._output_structure, ret)
   3366         return [ops.convert_to_tensor(t) for t in ret]

/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py in _wrapper_helper(*args)
   3297         nested_args = (nested_args,)
   3298 
-> 3299       ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
   3300       # If `func` returns a list of tensors, `nest.flatten()` and
   3301       # `ops.convert_to_tensor()` would conspire to attempt to stack

/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    256       except Exception as e:  # pylint:disable=broad-except
    257         if hasattr(e, 'ag_error_metadata'):
--> 258           raise e.ag_error_metadata.to_exception(e)
    259         else:
    260           raise

NotImplementedError: in user code:

    <ipython-input-16-3d72fc8870c2>:15 augmentation  *
        img = np.array(img)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:848 __array__  **
        " a NumPy call, which is not supported".format(self.name))

    NotImplementedError: Cannot convert a symbolic Tensor (args_0:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (3 by maintainers)

Most upvoted comments

@mdanatg The way I had tensorflow brought into my script was by using the colab magic cmd %tensorflow-version 2.x. On both days that brought in version 2.3.0. Does that answer your question?

I will definitely heed your advice and create a bug for that. I’ll spend some time formulating my request and post it in the next week or so.

Thank you for your help!

bjarrell15 on Nov 17, 2020

Okay, so I’ve modified the function to remove any reference to numpy and it’s now running successfully. I’m now using tensorflow_addons, but I strongly dislike the fact that it can’t fill in empty values with reflected values when I rotate it. Nonetheless, it’s functional:

!pip install -U tensorflow-addons
import tensorflow_addons as tfa

def augmentation(img, msk):

  # tf.math package for graph ops.
  import tensorflow.math as Math

  # Create some random floats, which will be used in augmentation steps.
  tilt = tf.random.uniform(shape = [], minval = -30, maxval = 30, dtype = tf.float32)
  dx = tf.random.uniform(shape = [], minval = -5, maxval = 5, dtype = tf.float32)
  dy = tf.random.uniform(shape = [], minval = -5, maxval = 5, dtype = tf.float32)

  # Use TensforFlow-style if conditionals, used to flip image and mask.
  img = tf.cond(tilt > 0, lambda: tf.image.flip_left_right(img), lambda: tf.image.flip_up_down(img))
  msk = tf.cond(tilt > 0, lambda: tf.image.flip_left_right(msk), lambda: tf.image.flip_up_down(msk))

  # Rotate the image and mask.

  # Convert "tilt" to radians.
  toRads = Math.multiply(Math.divide(tilt,180),tf.constant(math.pi))

  img = tfa.image.rotate(img, toRads)
  msk = tfa.image.rotate(msk, toRads)
 
  # Affine transformation
  img = tfa.image.translate(img, [dx,dy], 'BILINEAR')
  msk = tfa.image.translate(msk, [dx,dy], 'BILINEAR')

  # Put back into a tuple.
  finalTuple = (img, msk)

  return finalTuple

bjarrell15 on Nov 17, 2020