tfjs: tfjs-node works, tfjs-node-gpu fails with: Resource exhausted: OOM when allocating tensor with shape[2048]

TensorFlow.js version

tfjs-node-gpu: 0.1.18

Browser version

NodeJS v8.12.0, Windows 10, CUDA 9

Describe the problem or feature request

I have code in python which blurs an image, it works on Windows in Python on GPU. I am trying to do the same from node, I have successfully converted the script totfjs-node and it works and produces exactly the same results. When I try to run tfjs-node-gpu I go OOM, with the following error: Resource exhausted: OOM when allocating tensor with shape[2048] and type float log.txt

Command used to convert my model: Miniconda3\Scripts\tensorflowjs_converter.exe --input_format=tf_frozen_model --output_node_names="image_tensor,detection_boxes,detection_scores,detection_classes,num_detections" --saved_model_tags=serve frozen_inference_graph.pb web_model

Weights: weights_manifest.json.txt

Sample code in python: sample.py.txt

Sample code in js: sample.js.txt

What I have tried:

Tried with a very small input image, still failed.

Thoughts: Could this be a tensorflowjs_converter problem, where it expects the model to be ran in the browser, and converts to more generic types like int32 to store the data where the original code in python uses uint8.

This is how I pass the image to the model: tf.tensor4d(values, outShape, 'int32');, as I understand there is no uint8 support ? also for some reason this operation takes ~40ms, is this expected ?

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 15 (6 by maintainers)

Most upvoted comments

I’m running into the same issue. I’m running a frozen model for inference on a video stream. It works for about 2 minutes then I get a crash. It seems always to be after a period of time regardless of the inputs. The inference graph processes around 15 images per seconds.

Here is a relevent part (I hope) of my error:

Stats: 
Limit:                  7380579124
InUse:                  7380574720
MaxInUse:               7380574720
NumAllocs:                  856032
MaxAllocSize:            280559616
2018-10-28 14:32:12.367844: W tensorflow/core/common_runtime/bfc_allocator.cc:275] ****************************************************************************************************
2018-10-28 14:32:12.367860: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[1,187,103,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

It works with tfjs-node instead of tfjs-node-gpu (0.1.19).

I’m using a GeForce GTX 1070

hypersolution1 on Oct 28, 2018