realtime_object_detection: Internal: CUDA runtime implicit initialization on GPU:0 failed. Status: unknown error

Hello GustavZ, I ran into some problems running your code on the Jetson TX2. At first no problems at all but after a few days I keep receiving this error. full terminal log:

Model found. Proceed.
Loading frozen model into memory
2018-02-14 12:34:03.208044: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:881] could not open file to read NUMA node: /sys/bus/pci/devices/0000:00:00.0/numa_node
Your kernel may have been built without NUMA support.
2018-02-14 12:34:03.208210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties: 
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.66GiB freeMemory: 4.71GiB
2018-02-14 12:34:03.208272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-02-14 12:34:04.632828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:859] Could not identify NUMA node of /job:localhost/replica:0/task:0/device:GPU:0, defaulting to 0.  Your kernel may not have been built with NUMA support.
Loading label map
Starting detection
2018-02-14 12:34:28.283994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-02-14 12:34:28.284142: E tensorflow/core/common_runtime/direct_session.cc:168] Internal: CUDA runtime implicit initialization on GPU:0 failed. Status: unknown error
Traceback (most recent call last):
  File "object_detection.py", line 249, in <module>
    main()
  File "object_detection.py", line 245, in main
    detection(graph, category, score, expand)
  File "object_detection.py", line 170, in detection
    with tf.Session(graph=detection_graph,config=config) as sess:
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1509, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 628, in __init__
    self._session = tf_session.TF_NewDeprecatedSession(opts, status)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: Failed to create session.

If I first run the program without splitting the model and after wards again with the split turned on, it all works fine! But after I reboot, the problem arises again… Do you have any idea?

About this issue

Original URL
State: open
Created 6 years ago
Comments: 19 (4 by maintainers)

Most upvoted comments

I am still getting this issue on version 2.0.0

+29

sharad18 on Jan 17, 2020

I’m using the latest version of tensorflow (2.3.0) with python 3.6.10 and cuda 10.1 and facing the same issue as well in Ubuntu 18.04. export CUDA_VISIBLE_DEVICES=0 or export CUDA_VISIBLE_DEVICES=‘’ are helping to run the code but not consistent enough. I’m new and don’t actually know what these exports are doing exactly. This is the error I get: RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid

ctsams9 on Jul 31, 2020

Hi @GustavZ,

I searched about multiple session problem. here: https://devtalk.nvidia.com/default/topic/1035884/jetson-tx2/cuda-error-creating-more-than-one-session-using-tensorflow/post/5265161/#5265161

We need to add gpu_options in the tf.Session() that is called at the first. v1.0: object_detection.py v2.0: rod/model.py

def load_frozenmodel():
...
        input_graph = tf.Graph()
        config = tf.ConfigProto()
        config.gpu_options.allow_growth = allow_memory_growth
        with tf.Session(graph=input_graph, config=config):

naisy on Jun 12, 2018

@GustavZ now i install tensorflow1.7 on jetson tx2 but face same issue:

Loading label map Building Graph 2018-03-29 01:16:27.067350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0 2018-03-29 01:16:27.067445: E tensorflow/core/common_runtime/direct_session.cc:167] Internal: CUDA runtime implicit initialization on GPU:0 failed. Status: unknown error Traceback (most recent call last): File “object_detection.py”, line 301, in <module> main() File “object_detection.py”, line 297, in main detection(graph, category, score, expand) File “object_detection.py”, line 180, in detection with tf.Session(graph=detection_graph,config=config) as sess: File “/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py”, line 1509, in init super(Session, self).init(target, graph, config=config) File “/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py”, line 638, in init self._session = tf_session.TF_NewDeprecatedSession(opts, status) File “/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py”, line 516, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InternalError: Failed to create session.

can you suggest how to do?

imxboards on Mar 29, 2018

Okey, downgrading tensorflow to version 2.2 did it.

pip install --force-reinstall tensorflow-gpu==2.2

Also, if you have ever used pip install with --ignore-installed to install tensorflow versions or dependencies, consider removing them first.

What’s strange for me is that it takes some minutes to initialize tensorflow with GPU. I don’t think it’s normal :S

marcosrdac on Aug 13, 2020

Limiting the GPU memory solves this issue for me:

import tensorflow as tf

MEMORY_LIMIT = 1024
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
    try:
        tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=MEMORY_LIMIT)])
    except RuntimeError as e:
        print(e)

melvincabatuan on Apr 21, 2020