bitsandbytes: CUDA Setup failed despite GPU being available

I’m getting this error: CUDA Setup failed despite GPU being available

Even though I checked my Cuda installation with the provided nvidia cuda samples: https://github.com/NVIDIA/cuda-samples, and it passed all the tests. nvidia-smi and torch can see the GPU

bitsandbytes version

(venv) amrakm@amrakm:~/repos/EveryDream2trainer/bitsandbytes$ pip show bitsandbytes
Name: bitsandbytes
Version: 0.39.1
Summary: k-bit optimizers and matrix multiplication routines.
Home-page: https://github.com/TimDettmers/bitsandbytes
Author: Tim Dettmers
Author-email: dettmers@cs.washington.edu
License: MIT
Location: /home/amrakm/repos/EveryDream2trainer/venv/lib/python3.10/site-packages/bitsandbytes-0.39.1-py3.10.egg
Requires: 
Required-by: 

PATH and LD_LIBRARY_PATH variables:

(venv) amrakm@amrakm:~/repos/EveryDream2trainer/bitsandbytes$ CUDA_VERSION=118 make cuda11x
mkdir -p build
mkdir -p dependencies
ENVIRONMENT
============================
CUDA_VERSION: 118
============================
NVCC path: /usr/local/cuda-11.8/bin/nvcc
GPP path: /usr/bin/g++ VERSION: g++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
CUDA_HOME: /usr/local/cuda-11.8
CONDA_PREFIX: 
PATH: /home/amrakm/repos/EveryDream2trainer/venv/bin:/usr/local/cuda-11.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
LD_LIBRARY_PATH: /usr/local/cuda-12.2/lib64
============================
/usr/local/cuda-11.8/bin/nvcc -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86 -Xcompiler '-fPIC' --use_fast_math -Xptxas=-v -dc /home/amrakm/repos/EveryDream2trainer/bitsandbytes/csrc/ops.cu /home/amrakm/repos/EveryDream2trainer/bitsandbytes/csrc/kernels.cu -I /usr/local/cuda-11.8/include -I /home/amrakm/repos/EveryDream2trainer/bitsandbytes/csrc -I /include -I /home/amrakm/repos/EveryDream2trainer/bitsandbytes/include -L /usr/local/cuda-11.8/lib64 -lcudart -lcublas -lcublasLt -lcusparse -L /lib --output-directory /home/amrakm/repos/EveryDream2trainer/bitsandbytes/build
ptxas info    : 15 bytes gmem
ptxas info    : Compiling entry function '_ZN3cub11EmptyKernelIvEEvv' for 'sm_75'
ptxas info    : Function properties for _ZN3cub11EmptyKernelIvEEvv
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 4 registers, 352 bytes cmem[0]
ptxas info    : 15 bytes gmem
ptxas info    : Compiling entry function '_ZN3cub11EmptyKernelIvEEvv' for 'sm_80'
ptxas info    : Function properties for _ZN3cub11EmptyKernelIvEEvv

Error

(venv) amrakm@amrakm:~/repos/EveryDream2trainer/bitsandbytes$ python -m bitsandbytes

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/libbitsandbytes_cpu.so
False
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda-12.2/lib64')}
  warn(msg)
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/cuda-12.2/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('@/tmp/.ICE-unix/2170,unix/amrakm'), PosixPath('local/amrakm')}
  warn(msg)
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/etc/xdg/xdg-ubuntu')}
  warn(msg)
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('0'), PosixPath('1')}
  warn(msg)
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/org/gnome/Terminal/screen/a8a3cf82_72de_4f70_adea_1a373a8e49da')}
  warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
  warn(msg)
CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine!
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library...
  warn(msg)
CUDA SETUP: Required library version not found: libbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...

================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
1. CUDA driver not installed
2. CUDA not installed
3. You have multiple conflicting CUDA libraries
4. Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version via `conda list | grep cuda`.
================================================================================

CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected.
CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig.
CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following:
CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null
CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a
CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc
CUDA SETUP: Setup Failed!
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/__init__.py", line 6, in <module>
    from . import cuda_setup, utils, research
  File "/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/research/__init__.py", line 1, in <module>
    from . import nn
  File "/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/research/nn/__init__.py", line 1, in <module>
    from .modules import LinearFP8Mixed, LinearFP8Global
  File "/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/research/nn/modules.py", line 8, in <module>
    from bitsandbytes.optim import GlobalOptimManager
  File "/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/optim/__init__.py", line 6, in <module>
    from bitsandbytes.cextension import COMPILED_WITH_CUDA
  File "/home/amrakm/repos/EveryDream2trainer/bitsandbytes/bitsandbytes/cextension.py", line 20, in <module>
    raise RuntimeError('''
RuntimeError: 
        CUDA Setup failed despite GPU being available. Please run the following command to get more information:

        python -m bitsandbytes

        Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
        to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
        and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

---

(venv) amrakm@amrakm:~/repos/EveryDream2trainer/bitsandbytes$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print("CUDA Available: ", torch.cuda.is_available())
CUDA Available:  True

Mon Jul 10 12:49:11 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA Graphics...  On   | 00000000:09:00.0  On |                  Off |
| 30%   47C    P5    17W / 450W |    696MiB / 24564MiB |     13%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1923      G   /usr/lib/xorg/Xorg                346MiB |
|    0   N/A  N/A      2238      G   /usr/bin/gnome-shell               62MiB |
|    0   N/A  N/A     26210      G   ...841155447903966382,262144      161MiB |
|    0   N/A  N/A     29953      G   ...RendererForSitePerProcess      101MiB |
+-----------------------------------------------------------------------------+