tensorflow: java.lang.IllegalArgumentException: Internal error: Failed to apply delegate: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Nokia 7.1
TensorFlow installed from (source or binary): - TensorFlow version (use command below): tf-nightly 2.2-dev
Python version: - Bazel 3.7 version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: - GPU model and memory: Mobile gpu Adreno 506

You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior

I am trying to use the following model code here and have made a tensorflow lite model, when we run it on android we get the following error using the GPU delegate:

java.lang.IllegalArgumentException: Internal error: Failed to apply delegate: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors.

And when we switch to the CPU version we get the following error:


 Cannot convert between a TensorFlowLite tensor with type FLOAT32 and a Java object of type [I (which is compatible with the TensorFlowLite type INT32).
------------------------------------------------
whole error -> 2020-03-30 12:26:15.613 3471-3471/com.valuepitch.intruderdetector W/System.err: java.lang.IllegalArgumentException: Cannot convert between a TensorFlowLite tensor with type FLOAT32 and a Java object of type [I (which is compatible with the TensorFlowLite type INT32).
2020-03-30 12:26:15.614 3471-3471/com.valuepitch.intruderdetector W/System.err:     at org.tensorflow.lite.Tensor.throwIfTypeIsIncompatible(Tensor.java:316)
2020-03-30 12:26:15.614 3471-3471/com.valuepitch.intruderdetector W/System.err:     at org.tensorflow.lite.Tensor.getInputShapeIfDifferent(Tensor.java:218)
2020-03-30 12:26:15.614 3471-3471/com.valuepitch.intruderdetector W/System.err:     at org.tensorflow.lite.NativeInterpreterWrapper.run(NativeInterpreterWrapper.java:137)
2020-03-30 12:26:15.614 3471-3471/com.valuepitch.intruderdetector W/System.err:     at org.tensorflow.lite.Interpreter.runForMultipleInputsOutputs(Interpreter.java:311)

Describe the expected behavior

I expect the model to take 3 arguments and ouput the model outputs. I’ve seen similar issue here , but couldn’t comment there for unknown reasons or might be because the issue was closed? No clue as to that.

Standalone code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/Jupyter/any notebook.

Pls find colab notebook here

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 62 (23 by maintainers)

Most upvoted comments

@pranshugo maybe you could try to use tf.image.resize instead of keras.layers.Upsampling2D. In TF2.3, I encountered a same problem which I solved by using the above method.

zldrobit on Sep 17, 2020

Hey @sirius0503 , apologies for the delay.

Can you explain the resizing graphs with any frequency and the fixed cost and the associated shaders part part in a more simple manner and/or provide some links where I can read up more on them.

Sure! The short answer is that, when you use GPU delegation (or NNAPI delegation), there is a preparation stage where the graph is effectively recompiled into a different format, based on the current tensor shapes in the graph. For the GPU, this means creating and compiling a sequence of GPU shaders. For NNAPi, this means creating an NNAPI model, underneath which there can be additional preparation/processing done by various NNAPi drivers. All of this preparation can be expensive in terms of latency, often more expensive than a single inference call.

Currently, when an input is resized, all of that preparation work has to be executed again based on the new graph shapes. In an ideal world, the hardware backends could accommodate dynamic shapes, avoiding the need to “recompile” the model after a resize. However, implementing this behavior is complicated.

I can’t give a concrete timeline as to when this will be possible. Some workarounds that clients have used:

Create several different interpreters with various batch sizes, and use the right one depending on the current batch input.
Use a larger batch size than may be needed, always running with that batch size, but only feed in a smaller number of batches if necessary during inference.

jdduke on Jun 4, 2020

That worked out, Thanks @zldrobit Also I found out that tf.keras.Layers.Upsampling2D with interpolation argument as ‘nearest’ isn’t supported by tflite, whereas ‘bilnear’ works.

pranshugo on Sep 18, 2020

@monadiconion Your model’s input shape has an ‘unknown’ batch dim (The None above). If you can build the Keras model such that it has a fixed input shape (maybe by setting batch=1), the model will get acceleration support. Delegate backends work with fixed-shape graphs only, to avoid constant re-compilation of internal data structures.

@pranshugo Which version of TF/tflite_convert are you using for your conversion? Your model contains a ResizeNearestNeighbor op, which is causing the issue. I think its being used in a way not supported by TFLite. Can you post the snippet of code where its coming from?

srjoglekar246 on Sep 16, 2020

@monadiconion Your model’s input shape has an ‘unknown’ batch dim (The None above). If you can build the Keras model such that it has a fixed input shape (maybe by setting batch=1), the model will get acceleration support. Delegate backends work with fixed-shape graphs only, to avoid constant re-compilation of internal data structures.

@srjoglekar246 Almost all the pre-trained models have such kind of placeholder for batch dim, and some may even have other dynamic dims like width and height. In fact, most of the immediate layer’s output shape are dependent on the input size. Is there any way to set all the layers with a corresponding fixed value other than rebuild the model from scratch? It seems no API in TensorFlow can do it directly. How did you solve it when generated hosted models from pre-trained models in TensorFlow Hub? Thanks.

objectc on Oct 2, 2020

I’m having a similar problem.

Internal error: Failed to run on the given Interpreter: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors.

Though I’ve used static shapes like input1: (1, 256, 256, 3) and input2: (1, 256, 256, 3) while converting the model (using concrete function) to TFLite using both [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS] ops.

So, Question: How do you check whether the TFlite file has static sizes or dynamic sizes and which of the nodes/tensors have such dynamic sizes?

Misc Info: Using Tensorflow version 2.2 binary from Anaconda

PS: Can’t post the TFlite model due to company restrictions.

Thanks in advance!

tastelessjolt on Jun 11, 2020