onnxruntime: CUDAExecutionProvider Not Available / GPU Not Visible on NVIDIA T4

Describe the bug Do not see CUDAExecutionProvider or GPU available from ONNX Runtime even though onnxruntime-gpu is installed.

Urgency In critical stage of project & hence urgent.

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux lab-am-vm 4.19.0-16-cloud-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
ONNX Runtime installed from (source or binary): PIP Install
ONNX Runtime version: onnx==1.9.0 onnxruntime==1.7.0 onnxruntime-gpu==1.7.0
Python version: 3.7.10
CUDA/cuDNN version: release 11.0, V11.0.194
GPU model and memory: NVIDIA T4, 16G

To Reproduce

Describe steps/code to reproduce the behavior.

import onnxruntime as ort

ort.get_device()
>>> 'CPU'

ort.get_available_providers()
>>> ['CPUExecutionProvider']

Force setting code to CUDAExecutionProvider results in obvious error & inline with above telemetry:

import onnxruntime as ort

model_path = 'models/bart_large_cnn_fl/encoder/model.onnx'

providers = [
    ('CUDAExecutionProvider', {
        'device_id': 0,
        'arena_extend_strategy': 'kNextPowerOfTwo',
        'cuda_mem_limit': 2 * 1024 * 1024 * 1024,
        'cudnn_conv_algo_search': 'EXHAUSTIVE',
        'do_copy_in_default_stream': True,
    }),
    'CPUExecutionProvider',
]

session = ort.InferenceSession(model_path, providers=providers)

Error MSG

Expected behavior Expecting device as GPU and provider as CUDAExecutionProvider

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 17 (5 by maintainers)

Most upvoted comments

I have the same issue, using nvidia container nvcr.io/nvidia/pytorch:22.08-py3

Installing pip install onnxruntime-gpu==1.12 fixed the issue!

[EDIT] it looks like it’s random, it’s not always working

+21

FrancescoSaverioZuppichini on Sep 15, 2022

In my case the following helped:

uninstall onnxruntime
uninstall onnxruntime-gpu
install optimum[onnxruntime-gpu]

more here

thawro on Jun 13, 2023

it works for me

pip uninstall onnxruntime onnxruntime-gpu
pip install onnxruntime-gpu

takahashi-shotaro on Dec 22, 2023

Did you install both onnxruntime and onnxruntime_gpu on your machine ? If so, can you try after uninstalling onnxruntime with pip uninstall onnxruntime ?

hariharans29 on May 18, 2021

@hariharans29 It worked!! I guess the issue was below :

I was installing onnx, onnxruntime and onnxruntime-gpu in this sequence and later uninstalled onnxruntime only which created import issues. Instead, I uninstalled onnx as well this time and just installed onnxruntime-gpu from sctrach and that made it work, so for some reason onnx was conflicting the libs it seems …

Thank you so much for helping me get to this stage 😃

I see one issue though with this new method, one of the imports for quantize libs still fail, listed below - any suggestions on how to resolve this ?

anshoomehra on May 19, 2021

I’m having the same error, and I have never installed onnxruntime and onnx but only installed onnxruntime-gpu. I also checked the onnxruntime-gpu to CUDA compatibility. This is on latest version of Colab.

import onnxruntime

providers = onnxruntime.get_available_providers()
print(providers)

#  ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']

and I didn’t even notice that the session wasn’t utilizing the GPU until benchmarking against CPU, so I did:

X_ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(input_data, 'cuda', 0)

and started getting:

RuntimeError: /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1554 onnxruntime::ProviderInfo_CUDA& onnxruntime::GetProviderInfo_CUDA() CUDA Provider not available, can't get interface for it

merveenoyan on Feb 1, 2024

You don’t need onnx/onnxruntime installed if you plan to use onnxruntime_gpu.

Can you uninstall everything and just install onnxruntime_gpu ?

hariharans29 on May 18, 2021

@hariharans29 yes, I did uninstall and reinstall everything clean still the same issue. Sequence of install onnx, onnxruntime, onnxruntime-gpu…

onnx==1.9.0 onnxruntime==1.7.0 onnxruntime-gpu==1.7.0

I also just kept onnxruntime-gpu, uninstalling onnxruntime and import fails …

anshoomehra on May 18, 2021