tensorflow: dataset windows don't work in batches
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow):yes (see below)
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Windows 10 64 bit
- TensorFlow installed from (source or binary): pip install tensorflow, pip install tensorflow-nightly
- TensorFlow version (use command below):Currently on nightly - v1.12.1-43485-g0af213f96c 2.4.0-dev20201012 , but also happens on 2.3 release.
- Python version: Python 3.8.5 (default, Sep 3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc.
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version: Tried 10.1 and 11.0
- GPU model and memory: GeForce RTX 2080 with Max-Q Design computeCapability: 7.5 coreClock: 1.23GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 357.69GiB/s
Describe the current behavior
If I create a dataset of windowed data, then batch it up using tf.data.Dataset.window(5,1,1,True).batch(10,True) the output of iterating the resulting dataset doesn’t evaluate as a tensor and I can’t run model.fit() on it. It throws an exception:
TypeError: Inputs to a layer should be tensors. Got: <tensorflow.python.data.ops.dataset_ops._NestedVariant object at 0x000001BC82925250>
Describe the expected behavior
This not working basically breaks the tf.data loading mechanism for windowed data for me. I want to be able to create a preprocessing pipeline that does the following:
- load some data
- apply a sliding window to the data
- batch the data
- prefetch the data
- run model.fit
Standalone code to reproduce the issue
import tensorflow as tf
# make a source dataset
source_data=tf.data.Dataset.range(100)
# take two windows of different size onto the source dataset as
# our inputs and outputs
# and batch them up into 20 data points
inputs=source_data.window(10,1,1,True).batch(20,True)
targets=source_data.window(5,1,1,True).batch(20,True)
# zip this into a single training data set
training_set=tf.data.Dataset.zip((inputs,targets))
# a very simple network that takes input of [batch_size,10] and outputs [batch_size,5]
inLayer=tf.keras.layers.Input(shape=(10,),batch_size=20)
outLayer=tf.keras.layers.Dense(5)(inLayer)
model=tf.keras.Model(inputs=inLayer,outputs=outLayer)
model.compile(optimizer='adam',loss='mse')
# this fit call fails
model.fit(training_set)
Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Traceback (most recent call last):
File "d:/machinelearning/predictiveaudio/hellogoodbye/testtf.py", line 20, in <module>
model.fit(training_set)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1081, in fit
tmp_logs = self.train_function(iterator)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\def_function.py", line 828, in __call__
result = self._call(*args, **kwds)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\def_function.py", line 871, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\def_function.py", line 725, in _initialize
self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\function.py", line 2976, in _get_concrete_function_internal_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\function.py", line 3371, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\function.py", line 3206, in _create_graph_function
func_graph_module.func_graph_from_py_func(
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\framework\func_graph.py", line 990, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\eager\def_function.py", line 634, in wrapped_fn
out = weak_wrapped_fn().__wrapped__(*args, **kwds)
File "C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\framework\func_graph.py", line 977, in wrapper
raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\training.py:790 train_function *
return step_function(self, iterator)
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\training.py:780 step_function **
outputs = model.distribute_strategy.run(run_step, args=(data,))
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:1268 run
return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:2734 call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\distribute\distribute_lib.py:3355 _call_for_each_replica
return fn(*args, **kwargs)
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\training.py:773 run_step **
outputs = model.train_step(data)
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\training.py:739 train_step
y_pred = self(x, training=True)
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\base_layer.py:989 __call__
input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
C:\Users\micro\miniconda3\envs\tf23\lib\site-packages\tensorflow\python\keras\engine\input_spec.py:201 assert_input_compatibility
raise TypeError('Inputs to a layer should be tensors. Got: %s' % (x,))
TypeError: Inputs to a layer should be tensors. Got: <tensorflow.python.data.ops.dataset_ops._NestedVariant object at 0x000001BC82925250>
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 16 (3 by maintainers)
The batches should contain a series of values which are a sliding window over the underlying dataset. i.e. if I have a dataset [1,2,3,4,5,6,7,8,9,10], window size 3, batch size 2, I want tensors to come out like: [ [ 1,2,3],[2,3,4] ] [ [3,4,5],[4,5,6] ]
This is surely what window is for? It is to transform a dataset into another dataset, just like batch is? I can’t see what window is useful for if it doesn’t work to window time series data.
Essentially, it seems like when you are producing datasets using dataset transformations, a basic minimum expectation is that when you transform a dataset, the resulting transformed dataset should itself be able to generate input to model.fit?
Maybe it needs a different window method, but realistically, there’s no reason why it shouldn’t be possible to have a transformation which is used by model.fit which goes from a collection of N datasets of size AxB to tensors in the form NxAxB, without having to do flat_map on the structure.
The horrible workaround is to do a flat_map then batch twice. Which at least on my dataset seems slow.