AutoGPTQ: [BUG] RuntimeError: The detected CUDA version (12.1) mismatches the version that was used to compile PyTorch (11.8). Please make sure to use the same CUDA versions.

Describe the bug I’m attempting to install auto-gptq into an environment with:

pip install auto-gptq

The error I get is the following:

File "C:\Users\domin\anaconda3\envs\local-gpt\Lib\site-packages\torch\utils\cpp_extension.py", line 387, in _check_cuda_version
          raise RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
      RuntimeError:
      The detected CUDA version (12.1) mismatches the version that was used to compile
      PyTorch (11.8). Please make sure to use the same CUDA versions.
     
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for auto-gptq
  Running setup.py clean for auto-gptq
Failed to build auto-gptq
ERROR: Could not build wheels for auto-gptq, which is required to install pyproject.toml-based projects

I haven’t been able to use my environment with dev or nightly pytorch versions compatible with cuda 12.1 so I was hoping to fix this issue on the AutoGPTQ side. I’ve checked the nvidia website at https://www.nvidia.com/download/find.aspx looking for an earlier driver version I could downgrade to that would work with cuda 11.8, but the earliest supported is 527.56, which according to https://docs.nvidia.com/deploy/cuda-compatibility/index.html is not compatible with cuda 11.8. I’m not entirely sure how the versioning works though, because despite this my pytorch version compiled with cuda 11.8 works fine with my 12.1 drivers, so I’m unsure why Auto-GPTQ doesn’t.

Hardware details My nvidia gpu and drivers are:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98                 Driver Version: 535.98       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2070 ...  WDDM  | 00000000:29:00.0  On |                  N/A |
|ERR!    0C    P0             ERR! / 235W |   1071MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

Software version The version of autogptq pip is attempting to install is:

Downloading auto_gptq-0.2.2.tar.gz (52 kB)

I have also encountered the same issue with:

auto-gptq 0.2.2+cu118

I have the following version of pytorch installed:

# Name                    Version                   Build  Channel
torch                     2.0.1+cu118              pypi_0    pypi
torchvision               0.15.2+cu118             pypi_0    pypi

I have cudatoolkit 11.8 installed:

# Name                    Version                   Build  Channel
cudatoolkit               11.8.0               hd77b12b_0

My python version is:

# Name                    Version                   Build  Channel
python                    3.11.3               he1021f5_1

An exhaustive list of all of the packages installed in the environment is:

# Name                    Version                   Build  Channel
accelerate                0.20.3                   pypi_0    pypi
aiohttp                   3.8.4                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
anyio                     3.7.0                    pypi_0    pypi
async-timeout             4.0.2                    pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
backoff                   2.2.1                    pypi_0    pypi
bitsandbytes              0.39.0                   pypi_0    pypi
bzip2                     1.0.8                he774522_0
ca-certificates           2023.05.30           haa95532_0
certifi                   2023.5.7                 pypi_0    pypi
cffi                      1.15.1                   pypi_0    pypi
charset-normalizer        3.1.0                    pypi_0    pypi
chromadb                  0.3.22                   pypi_0    pypi
click                     8.1.3                    pypi_0    pypi
clickhouse-connect        0.6.2                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
cryptography              41.0.1                   pypi_0    pypi
cudatoolkit               11.8.0               hd77b12b_0
dataclasses-json          0.5.7                    pypi_0    pypi
datasets                  2.13.0                   pypi_0    pypi
dill                      0.3.6                    pypi_0    pypi
duckdb                    0.8.0                    pypi_0    pypi
et-xmlfile                1.1.0                    pypi_0    pypi
faiss-cpu                 1.7.4                    pypi_0    pypi
fastapi                   0.96.0                   pypi_0    pypi
filelock                  3.12.1                   pypi_0    pypi
frozenlist                1.3.3                    pypi_0    pypi
fsspec                    2023.6.0                 pypi_0    pypi
greenlet                  2.0.2                    pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
hnswlib                   0.7.0                    pypi_0    pypi
httptools                 0.5.0                    pypi_0    pypi
huggingface-hub           0.15.1                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
instructorembedding       1.0.1                    pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
joblib                    1.2.0                    pypi_0    pypi
langchain                 0.0.191                  pypi_0    pypi
libffi                    3.4.4                hd77b12b_0
llama-cpp-python          0.1.48                   pypi_0    pypi
lz4                       4.3.2                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
marshmallow               3.19.0                   pypi_0    pypi
marshmallow-enum          1.5.1                    pypi_0    pypi
monotonic                 1.6                      pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.0.4                    pypi_0    pypi
multiprocess              0.70.14                  pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
networkx                  3.1                      pypi_0    pypi
nltk                      3.8.1                    pypi_0    pypi
numexpr                   2.8.4                    pypi_0    pypi
numpy                     1.24.3                   pypi_0    pypi
openapi-schema-pydantic   1.2.4                    pypi_0    pypi
openpyxl                  3.1.2                    pypi_0    pypi
openssl                   3.0.8                h2bbff1b_0
packaging                 23.1                     pypi_0    pypi
pandas                    2.0.2                    pypi_0    pypi
pdfminer-six              20221105                 pypi_0    pypi
pillow                    9.5.0                    pypi_0    pypi
pip                       23.1.2          py311haa95532_0
posthog                   3.0.1                    pypi_0    pypi
protobuf                  3.20.0                   pypi_0    pypi
psutil                    5.9.5                    pypi_0    pypi
pyarrow                   12.0.1                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pydantic                  1.10.9                   pypi_0    pypi
python                    3.11.3               he1021f5_1
python-dateutil           2.8.2                    pypi_0    pypi
python-dotenv             1.0.0                    pypi_0    pypi
pytz                      2023.3                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
regex                     2023.6.3                 pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
rouge                     1.0.1                    pypi_0    pypi
safetensors               0.3.1                    pypi_0    pypi
scikit-learn              1.2.2                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
sentence-transformers     2.2.2                    pypi_0    pypi
sentencepiece             0.1.99                   pypi_0    pypi
setuptools                67.8.0          py311haa95532_0
six                       1.16.0                   pypi_0    pypi
sniffio                   1.3.0                    pypi_0    pypi
sqlalchemy                2.0.15                   pypi_0    pypi
sqlite                    3.41.2               h2bbff1b_0
starlette                 0.27.0                   pypi_0    pypi
sympy                     1.12                     pypi_0    pypi
tenacity                  8.2.2                    pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tk                        8.6.12               h2bbff1b_0
tokenizers                0.13.3                   pypi_0    pypi
torch                     2.0.1+cu118              pypi_0    pypi
torchvision               0.15.2+cu118             pypi_0    pypi
tqdm                      4.65.0                   pypi_0    pypi
transformers              4.30.1                   pypi_0    pypi
typing-extensions         4.6.3                    pypi_0    pypi
typing-inspect            0.9.0                    pypi_0    pypi
tzdata                    2023.3                   pypi_0    pypi
urllib3                   1.26.6                   pypi_0    pypi
uvicorn                   0.22.0                   pypi_0    pypi
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
watchfiles                0.19.0                   pypi_0    pypi
websockets                11.0.3                   pypi_0    pypi
wheel                     0.38.4          py311haa95532_0
xxhash                    3.2.0                    pypi_0    pypi
xz                        5.4.2                h8cc25b3_0
yarl                      1.9.2                    pypi_0    pypi
zlib                      1.2.13               h8cc25b3_0
zstandard                 0.21.0                   pypi_0    pypi

My OS is Windows 10 Pro version 22H2

Expected behavior AutoGPTQ installs successfully

About this issue

Original URL
State: open
Created a year ago
Reactions: 4
Comments: 15 (6 by maintainers)

Most upvoted comments

Incidentally @mcgeochd you could also try just directly installing a pre-built wheel with eg:

pip uninstall -y auto-gptq
pip install https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.2.2/auto_gptq-0.2.2+cu118-cp310-cp310-linux_x86_64.whl

That would install the AutoGPTQ 11.8 wheel for Python 3.10 on Linux. If you’re using a different version of Python, or are on Windows, then you’d swap out the URL for one of the others listed here: https://github.com/PanQiWei/AutoGPTQ/releases/tag/v0.2.2

I don’t know if that will work, but it’s worth a try. It will definitely install. The question will then be whether the extension will work.

But that’s the method that text-generation-webui uses for example. It just directly installs a AutoGPTQ wheel, thus bypassing the compilation process. And it seems to work for users of that UI.

TheBloke on Jun 18, 2023

I’m a bit confused. According to your nvidia-smi output you’re using an NVidia 2070? Which is an old GPU and could use a much older version of the NVidia drivers.

FYI your nvidia-smi is reporting some errors, which makes me wonder if you have some issue there:

What is your exact GPU model name, and could you please link to whatever you’re seeing suggesting that you can’t use an NV driver older than 527?

However regardless of that it shouldn’t be an issue anyway, because you definitely can use 11.8 with newer NV driver versions. Eg here I am using CUDA toolkit 11.8 with NV driver 530.30, with no problems at all:

 root@22437ad3eef0:/workspace ᐅ nvidia-smi
Sat Jun 17 20:09:40 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA L40                      On | 00000000:81:00.0 Off |                    0 |
| N/A   25C    P8               34W / 300W|      2MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
 root@22437ad3eef0:/workspace ᐅ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

I could be using (and have used) 11.6 even and it’d work.

So I’d suggest you install CUDA 11.8 with pytorch 2.0.1 and then everything will be fully supported.

PS if you really felt you had to use CUDA 12.x, and can’t use pytorch 2.1 nightly, then there is one more option: you can build pytorch 2.0.1 from source against CUDA 12.x. Here’s how I do that on Linux:

pip install mkl mkl-include cmake ninja
git clone https://github.com/pytorch/pytorch.git pytorch201
cd pytorch201
git checkout v2.0.1
git submodule update --init --recursive
pip install -r requirements.txt
python setup.py bdist_wheel
pip install dist/torch-2.0.0a0+gite9ebda2-cp310-cp310-linux_x86_64.whl

Instructions may differ on Windows, so if you’re on that platform and want to compile, check the pytorch docs.

TheBloke on Jun 17, 2023

In theory any 11.x or 12.x version is fine.

However pytorch currently only supports CUDA 11.x. You can use pytorch with 12.x, but you either need to use a nightly build of pytorch 2.1, or compile your own pytorch 2.0.1. AutoGPTQ works fine with pytorch 2.1 in my (brief) testing but not many people will be using that yet so it would be considered an untested and unsupported configuration.

Compiling pytorch 2.0.1 with CUDA 12.x support is another option, and I do that myself on one system. But compiling pytorch does take a while to do, and there’s a few steps to go through.

Therefore right now the recommended and supported version of CUDA is 11.8.

11.8 supports all the latest GPUs and to my knowledge there’s no performance difference between 12.x and 11.8, so I don’t know of any pressing reason to move off 11.8 until pytorch 2.1 officially releases.

TheBloke on Jun 20, 2023