tensorflow: TensorRT INT8 calibration doesn't work with TF r1.12 and TRT 5RC

Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn’t be a TensorBoard issue. Those go here.

Here’s why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):n/a
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: n/a
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): r1.12
Python version: 3.5
Bazel version (if compiling from source): 0.16.1
GCC/Compiler version (if compiling from source): 5.4.0
CUDA/cuDNN version:9/7.1
GPU model and memory:1080ti
Exact command to reproduce:

Follow workflow from here, https://devblogs.nvidia.com/tensorrt-integration-speeds-tensorflow-inference/ Error at the following piece of code trt_graph=trt.calib_graph_to_infer_graph(calibGraph) Log: File “/home/dhingratul/.virtualenvs/tf_trt_source_trt5rc_tf1_12/local/lib/python3.5/site-packages/tensorflow/contrib/tensorrt/python/trt_convert.py”, line 349, in calib_graph_to_infer_graph for n in calibration_graph_def.node: AttributeError: ‘Graph’ object has no attribute ‘node’

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c “import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)”

Describe the problem

Describe the problem clearly here. Be sure to convey here why it’s a bug in TensorFlow or a feature request.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 20 (6 by maintainers)

Most upvoted comments

@samikama Based on your recommendation, this is the workflow I have, but i am facing a weird CUBLAS issue now 2018-10-12 13:15:52.338802: E tensorflow/stream_executor/cuda/cuda_blas.cc:652] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_INTERNAL_ERROR

fid = "model.pb"
output_nodenames = 'op1,op2,op3'
output_node = list(output_nodenames.split(","))
g = load_graph(fid)
with tf.Session(graph=g) as sess:
	# Step 1 -- Create Inference graph
    trt_graph = trt.create_inference_graph(
    input_graph_def=tf.get_default_graph().as_graph_def(),
    outputs=output_node,
    max_batch_size=20000,
    max_workspace_size_bytes=1 << 25,
    precision_mode="INT8",  # TRT Engine precision "FP32","FP16" or "INT8"
    minimum_segment_size=2  # minimum number of nodes in an engine
    )
    # Step 2 -- Calibration
    tf.import_graph_def(
    trt_graph,
    name='', #DEBUG
	)
    num_samples = 10
    np.random.seed(0)
    ip1_data = np.random.rand(num_samples,700,800,6).astype(np.float32)
    ip1 = tf.get_default_graph().get_tensor_by_name("ip1:0")

    ip2_data = np.random.rand(4).astype(np.float32)
    ip2 = tf.get_default_graph().get_tensor_by_name("ip2:0")

    ip3_data = np.random.rand(20000,6).astype(np.float32)
    ip3 = tf.get_default_graph().get_tensor_by_name("ip3:0")

    ip4_data = np.random.rand(20000,4).astype(np.float32)
    ip4 = tf.get_default_graph().get_tensor_by_name("ip4:0")
    out1 = tf.get_default_graph().get_tensor_by_name("op1:0")
    out2 = tf.get_default_graph().get_tensor_by_name("op2:0")
    out3 = tf.get_default_graph().get_tensor_by_name("op3:0")

    for i in range(num_samples):
        start_time = timeit.default_timer()
        _ = sess.run([out1, out2, out3], feed_dict={ip1:ip1_data[i], ip2:ip2_data, ip3:ip3_data, ip4:ip4_data})
    # Step 3 -- pass through calib_graph_to_infer_graph
    trt_graph = trt.calib_graph_to_infer_graph(trt_graph)
    with tf.gfile.GFile("trt.pb", "wb") as f:
        f.write(trt_graph.SerializeToString())

dhingratul on Oct 12, 2018