numba: Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR

Hi there,

I am using numba without conda in Manjaro OS. That is, just using the nvidia driver and nvidia toolkit system installations. Numba was working ok until Manjaro updated the driver and the CUDA Toolkit in my system. Now I am getting the following error: “numba.cuda.cudadrv.driver.LinkerError: [222] Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR ptxas application ptx input, line 9; fatal : Unsupported .version 7.3; current version is ‘7.2’”

I have uninstalled numba with pip and then install it again and the error keeps coming.
I have run numba -s and I get no errors:

System info:
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2021-05-07 12:26:36.702571
UTC start time                                : 2021-05-07 16:26:36.702576
Running time (s)                              : 2.191629

__Hardware Information__
Machine                                       : x86_64
CPU Name                                      : ivybridge
CPU Count                                     : 8
Number of accessible CPUs                     : 8
List of accessible CPUs cores                 : 0 1 2 3 4 5 6 7
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : 64bit aes avx cmov cx16 cx8 f16c
                                                fsgsbase fxsr mmx pclmul popcnt
                                                rdrnd sahf sse sse2 sse3 sse4.1
                                                sse4.2 ssse3 xsave xsaveopt

Memory Total (MB)                             : 15958
Memory Available (MB)                         : 12346

__OS Information__
Platform Name                                 : Linux-5.9.16-1-MANJARO-x86_64-with-glibc2.33
Platform Release                              : 5.9.16-1-MANJARO
OS Name                                       : Linux
OS Version                                    : #1 SMP PREEMPT Mon Dec 21 22:00:46 UTC 2020
OS Specific Version                           : ?
Libc Version                                  : glibc 2.33

__Python Information__
Python Compiler                               : GCC 10.2.0
Python Implementation                         : CPython
Python Version                                : 3.9.4
Python Locale                                 : en_GB.UTF-8

__LLVM Information__
LLVM Version                                  : 10.0.1

__CUDA Information__
CUDA Device Initialized                       : True
CUDA Driver Version                           : 11020
CUDA Detect Output:
Found 1 CUDA devices
id 0    b'GeForce GTX 1070 Ti'                              [SUPPORTED]
                      compute capability: 6.1
                           pci device id: 0
                              pci bus id: 1
Summary:
	1/1 devices are supported

CUDA Libraries Test Output:
Finding cublas from CUDA_HOME
	named  libcublas.so.11.4.2.10064
	trying to open library...	ok
Finding cusparse from CUDA_HOME
	named  libcusparse.so.11.5.0.58
	trying to open library...	ok
Finding cufft from CUDA_HOME
	named  libcufft.so.10.4.2.58
	trying to open library...	ok
Finding curand from CUDA_HOME
	named  libcurand.so.10.2.4.58
	trying to open library...	ok
Finding nvvm from CUDA_HOME
	named  libnvvm.so.4.0.0
	trying to open library...	ok
Finding cudart from CUDA_HOME
	named  libcudart.so.11.3.58
	trying to open library...	ok
Finding cudadevrt from CUDA_HOME
	named  libcudadevrt.a
Finding libdevice from CUDA_HOME
	searching for compute_20...	ok
	searching for compute_30...	ok
	searching for compute_35...	ok
	searching for compute_50...	ok


__ROC information__
ROC Available                                 : False
ROC Toolchains                                : None
HSA Agents Count                              : 0
HSA Agents:
None
HSA Discrete GPUs Count                       : 0
HSA Discrete GPUs                             : None

__SVML Information__
SVML State, config.USING_SVML                 : False
SVML Library Loaded                           : False
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : False

__Threading Layer Information__
TBB Threading Layer Available                 : True
+-->TBB imported successfully.
OpenMP Threading Layer Available              : True
+-->Vendor: GNU
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
NUMBA_CUDA_DRIVER                             : /usr/lib/libcuda.so

__Conda Information__
Conda not available.

__Installed Packages__
Package             Version
------------------- -----------------
appdirs             1.4.4
apsw                3.35.4.post1
arandr              0.1.10
argcomplete         1.10.0
argon2-cffi         20.1.0
asn1crypto          1.4.0
astropy             4.2
astropy-healpix     0.5
astropy-helpers     4.0.1
astroquery          0.4.1
async-generator     1.10
attrs               20.3.0
autopep8            1.5.7
backcall            0.2.0
beautifulsoup4      4.8.0
bleach              3.3.0
blurhash            1.1.4
btrfsutil           5.11.1
CacheControl        0.12.6
cchardet            2.1.7
cffi                1.14.5
chardet             3.0.4
cloudpickle         1.6.0
colorama            0.4.4
configobj           5.1.0.dev0
contextlib2         0.6.0.post1
cryptography        3.4.7
css-parser          1.0.6
cssselect           1.1.0
cupy                8.6.0
cycler              0.10.0
dask                2021.4.1
dask-ms             0.2.6
decorator           5.0.7
defusedxml          0.7.1
distlib             0.3.1
distro              1.5.0
dnspython           1.16.0
docopt              0.6.2
docutils            0.16
docx2txt            0.8
EbookLib            0.17.1
entrypoints         0.3
extract-msg         0.23.1
fastrlock           0.6
feedparser          5.2.1
fsspec              2021.4.0
funcsigs            1.0.2
future              0.18.2
healpy              1.14.0
html2text           2020.1.16
html5-parser        0.4.9
html5lib            1.1
idna                3.1
ifaddr              0.1.7
imageio             2.9.0
IMAPClient          2.1.0
importlib-metadata  3.10.0
ipykernel           5.5.3
ipython             7.22.0
ipython-genutils    0.2.0
ipywidgets          7.6.3
isodate             0.6.0
JACK-Client         0.5.3
jedi                0.18.0
jeepney             0.6.0
Jinja2              2.11.3
joblib              1.0.1
jsonschema          3.2.0
jupyter-client      6.1.12
jupyter-console     6.4.0
jupyter-core        4.7.1
jupyterlab-pygments 0.1.2
keyring             23.0.1
kiwisolver          1.3.1
lic                 0.4.5
llvmlite            0.36.0
locket              0.2.1
louis               3.17.0
lxml                4.6.3
Mako                1.1.4
Markdown            3.3.4
MarkupSafe          1.1.1
Mastodon.py         1.5.1
matplotlib          3.3.4
mechanize           0.4.5
meld                3.20.3
mistune             0.8.4
msgpack             1.0.2
nbclient            0.5.3
nbconvert           6.0.7
nbformat            5.1.3
nest-asyncio        1.5.1
netifaces           0.10.9
netsnmp-python      1.0a1
notebook            6.3.0
npyscreen           4.10.5
numba               0.53.1
numpy               1.20.1
olefile             0.46
ordered-set         4.0.2
packaging           20.9
pacman-mirrors      4.21.2
palettable          3.3.0
pandas              1.2.4
pandocfilters       1.4.3
parso               0.8.1
partd               1.2.0
pdfminer.six        20181108
pep517              0.10.0
pexpect             4.8.0
pickleshare         0.7.5
Pillow              8.2.0
pip                 21.1.1
pkginfo             1.7.0
ply                 3.11
progress            1.5
prometheus-client   0.10.1
prompt-toolkit      3.0.18
protonvpn-cli       2.2.6
prox-tv             3.2.1
proxmin             0.6.11
psutil              5.8.0
ptyprocess          0.7.0
py7zr               0.11.3
pycairo             1.20.0
pychm               0.8.6
pycodestyle         2.7.0
pycountry           20.7.3
pycparser           2.20
pycryptodome        3.10.1
pyerfa              1.7.2
Pygments            2.8.1
PyGObject           3.40.1
pynufft             2021.0.2
pyopencl            2021.1.6
PyOpenGL            3.1.5
pyOpenSSL           20.0.1
pyparsing           2.4.7
PyQt5               5.15.4
PyQt5-sip           12.8.1
PyQtWebEngine       5.15.4
pyrsistent          0.17.3
PySide2             5.15.2
pysmbc              1.0.23
PySocks             1.7.1
python-casacore     3.4.0
python-dateutil     2.8.1
python-libtorrent   1.2.11
python-magic        0.4.18
python-pptx         0.6.18
python-xlib         0.29
pythondialog        3.5.1
pytools             2021.2.3
pytz                2021.1
pywal               3.3.0
PyYAML              5.4.1
pyzmq               22.0.3
ranger-fm           1.9.3
readme-renderer     29.0
regex               2021.4.4
regions             0.4
reikna              0.7.5
reportlab           3.5.66
reproject           0.7.1
requests            2.25.1
requests-toolbelt   0.9.1
resolvelib          0.5.4
retrying            1.3.3
rfc3986             1.4.0
scipy               1.6.1
seaborn             0.11.1
SecretStorage       3.3.1
Send2Trash          1.5.0
setuptools          54.2.0
shiboken2           5.15.2
six                 1.12.0
sortedcontainers    2.3.0
soupsieve           2.2.1
SpeechRecognition   3.8.1
speedtest-cli       2.1.3
SQLAlchemy          1.3.22
streamlink          2.0.0+22.gb9d0d18
team                1.0
terminado           0.9.4
terminator          2.1.1
testpath            0.4.4
textract            1.6.3
texttable           1.6.3
toml                0.10.2
toolz               0.11.1
tornado             6.1
tqdm                4.59.0
traitlets           5.0.5
twine               3.4.1
tzlocal             1.5.1
ueberzug            18.1.9
ufw                 0.36
unrardll            0.1.4
urllib3             1.26.4
validate            5.1.0.dev0
visvis              1.13.0
wcwidth             0.2.5
webencodings        0.5.1
websocket-client    0.58.0
wheel               0.36.2
widgetsnbextension  3.5.1
xarray              0.17.0
xlrd                1.2.0
XlsxWriter          1.3.9
zeroconf            0.28.8
zipp                3.4.1

No errors reported.


__Warning log__
Warning (roc): Error initialising ROC: No ROC toolchains found.
Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init: 
NUMBA_HSA_DRIVER /opt/rocm/lib/libhsa-runtime64.so is not a valid file path.  Note it must be a filepath of the .so/.dll/.dylib or the driver:
Warning: Conda not available.
 Error was [Errno 2] No such file or directory: 'conda'

--------------------------------------------------------------------------------

I will leave here the result of my nvcc and nvidia-smi commands:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0

Fri May  7 12:29:13 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 107...  Off  | 00000000:01:00.0  On |                  N/A |
|  0%   57C    P0    36W / 180W |    200MiB /  8116MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A       650      G   /usr/lib/Xorg                     144MiB |
|    0   N/A  N/A      1272      G   protonmail-bridge                   5MiB |
|    0   N/A  N/A      1425      G   ...java-11-openjdk//bin/java        7MiB |
|    0   N/A  N/A      2150      G   ...AAAAAAAAA= --shared-files       40MiB |
+-----------------------------------------------------------------------------+

Does anyone know how to fix this?

Cheers

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 19 (10 by maintainers)

Most upvoted comments

It looks like you have the CUDA Toolkit version 11.3 but only a driver version of 11.2. You will need to upgrade to a later driver of at least version 465.19.01, to match CUDA toolkit 11.3 (See Table 2 of https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html).

Note that although Forward Compatibility is a feature of the CUDA Toolkit 11.x series, which allows you to use code compiled with a newer toolkit than the current driver version, Numba presently doesn’t implement support for Forward Compatibility.

gmarkall on May 7, 2021

Thanks for the info. You’re using the CUDA 11.6 toolkit with a driver that only supports up to CUDA 11.5:

CUDA Driver Version                           : (11, 5)
CUDA Runtime Version                          : 11060

You need to either:

Update to a driver that supports CUDA 11.6 (510.39.01 or above), or
Downgrade to using the CUDA toolkit 11.5.

To use CUDA 11.5, I think you will be able to do something like:

conda install conda-forge::cudatoolkit=11.5

gmarkall on Jan 21, 2022

@gmarkall I recently encountered a similar problem when trying to make use of Forward Compatibility (driver 11.2, toolkit 11.3). It’s not trivial to me what’s the “support” you are referring to. Doesn’t it work out of box?

It works out of the box for CUDA C/C++ as far as I am aware - however, because Numba doesn’t know anything about forward compatibility it always tries to generate PTX for the latest version supported by the toolkit and not the driver, so the driver refuses to accept it for linking with the error message reported in this issue:

numba.cuda.cudadrv.driver.LinkerError: [222] Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR ptxas application ptx input, line 9; fatal : Unsupported .version 7.3; current version is '7.2'

gmarkall on May 10, 2021

@gmarkall thank you so much for your help, I really appreciate it! I saw that one too and was quite baffled/suspicious at the time…

Jolly, CUDA 11.6 runtime (CUDA Runtime Version 11060) somehow mysteriously crept onto my Ubuntu system (although, if my memory isn’t failing me, I explicitly chose not to upgrade this). Running sudo apt autoremove actually removed the impostor runtime and solved this issue for me, and now numba -s | grep "Runtime" produces CUDA Runtime Version : 11050.

Not using conda here, just plain old pip, and my tests using numba are now passing! Thank you!

Algomorph on Jan 21, 2022