exllamav2: CU12+ appears to be unsupported?

pip installing the JIT version fails so below I have pasted the error message for the build from source. I think a temporary workaround for me would be to grab a dockerfile that has CU11.8, because my rolling release distro updated cuda for me and I suspect that this is what made it stop working.

My system:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Sep__8_19:17:24_PDT_2023
Cuda compilation tools, release 12.3, V12.3.52
Build cuda_12.3.r12.3/compiler.33281558_0

$ uname -r
6.5.9-arch2-1

$ gcc --version
gcc (GCC) 13.2.1 20230801

$ gcc-12 --version
gcc-12 (GCC) 12.3.0

Error message build from source

python setup.py install --user
Version: 0.0.7
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
TEST FAILED: /home/seb/.local/lib/python3.11/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /home/seb/.local/lib/python3.11/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching 'dni_*' found anywhere in distribution
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'exllamav2.exllamav2_ext' as data is deprecated, please list it in `packages`.
    !!


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'exllamav2.exllamav2_ext' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'exllamav2.exllamav2_ext' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'exllamav2.exllamav2_ext' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.


!!

  check.warn(importable)
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'exllamav2.exllamav2_ext.cpp' as data is deprecated, please list it in `packages`.
    !!


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'exllamav2.exllamav2_ext.cpp' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'exllamav2.exllamav2_ext.cpp' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'exllamav2.exllamav2_ext.cpp' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.


!!

  check.warn(importable)
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'exllamav2.exllamav2_ext.cuda' as data is deprecated, please list it in `packages`.
    !!


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'exllamav2.exllamav2_ext.cuda' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'exllamav2.exllamav2_ext.cuda' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'exllamav2.exllamav2_ext.cuda' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.


!!

  check.warn(importable)
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'exllamav2.exllamav2_ext.cuda.quant' as data is deprecated, please list it in `packages`.
    !!


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'exllamav2.exllamav2_ext.cuda.quant' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'exllamav2.exllamav2_ext.cuda.quant' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'exllamav2.exllamav2_ext.cuda.quant' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.


!!

  check.warn(importable)
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'exllamav2.generator.filters' as data is deprecated, please list it in `packages`.
    !!


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'exllamav2.generator.filters' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'exllamav2.generator.filters' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'exllamav2.generator.filters' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.


!!

  check.warn(importable)
/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'exllamav2.server' as data is deprecated, please list it in `packages`.
    !!


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'exllamav2.server' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    'exllamav2.server' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that 'exllamav2.server' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.


!!

  check.warn(importable)
/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py:414: UserWarning: The detected CUDA version (12.3) has a minor version mismatch with the version that was used to compile PyTorch (12.1). Most likely this shouldn't be a problem.
  warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py:424: UserWarning: There are no g++ version bounds defined for CUDA version 12.3
  warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
Emitting ninja build file /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/rope.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/rope.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/rope.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/rope.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/rope.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/rope.cu".
[2/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/cache.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/cache.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/cache.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/cache.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/cache.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/cache.cu".
[3/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/rms_norm.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/rms_norm.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/rms_norm.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/rms_norm.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/rms_norm.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/rms_norm.cu".
[4/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/pack_tensor.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/pack_tensor.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/pack_tensor.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/pack_tensor.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/pack_tensor.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/pack_tensor.cu".
[5/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_matrix.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_matrix.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_matrix.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_matrix.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_matrix.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_matrix.cu".
[6/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/quantize.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/quantize.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/quantize.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/quantize.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/quantize.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/quantize.cu".
[7/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_gemm.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_gemm.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_gemm.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_gemm.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_gemm.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_gemm.cu".
[8/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_mlp.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_mlp.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_mlp.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_mlp.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_mlp.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_mlp.cu".
[9/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/lora.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/lora.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/lora.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/lora.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/lora.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/lora.cu".
[10/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/h_gemm.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/h_gemm.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/h_gemm.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/h_gemm.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/h_gemm.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/h_gemm.cu".
[11/11] /opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_attn.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_attn.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_attn.o 
/opt/cuda/bin/nvcc  -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -I/home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -I/opt/cuda/include -I/home/seb/.venvs/x/include -I/usr/include/python3.11 -c -c /home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_attn.cu -o /home/seb/Code/x/exllamav2/build/temp.linux-x86_64-cpython-311/exllamav2/exllamav2_ext/cuda/q_attn.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=exllamav2_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/Code/x/exllamav2/exllamav2/exllamav2_ext/cuda/q_attn.cu".
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/seb/Code/x/exllamav2/setup.py", line 57, in <module>
    setup(
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    self.run_command(cmd)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/bdist_egg.py", line 165, in run
    cmd = self.call_command('install_lib', warn_dir=0)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/bdist_egg.py", line 151, in call_command
    self.run_command(cmdname)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/command/install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    self.build_extensions()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 873, in build_extensions
    build_ext.build_extensions(self)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
    self._build_extensions_serial()
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 246, in build_extension
    _build_ext.build_extension(self, ext)
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 547, in build_extension
    objects = self.compiler.compile(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 686, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1774, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension

Errror message pip install JIT

$ python test.py 
Traceback (most recent call last):
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/ext.py", line 14, in <module>
    import exllamav2_ext
ModuleNotFoundError: No module named 'exllamav2_ext'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/seb/Code/x/test.py", line 1, in <module>
    from lib.llm.prepare import load
  File "/home/seb/Code/x/lib/llm/prepare.py", line 1, in <module>
    from exllamav2 import ExLlamaV2Config, ExLlamaV2, ExLlamaV2Tokenizer, ExLlamaV2Cache
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/__init__.py", line 3, in <module>
    from exllamav2.model import ExLlamaV2
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/model.py", line 11, in <module>
    from exllamav2.cache import ExLlamaV2Cache
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/cache.py", line 2, in <module>
    from exllamav2.ext import exllamav2_ext as ext_c
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/ext.py", line 124, in <module>
    exllamav2_ext = load \
                    ^^^^^^
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
           ^^^^^^^^^^^^^
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/seb/.venvs/x/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'exllamav2_ext': [1/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/pack_tensor.cu -o pack_tensor.cuda.o 
FAILED: pack_tensor.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/pack_tensor.cu -o pack_tensor.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/pack_tensor.cu".
[2/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/rms_norm.cu -o rms_norm.cuda.o 
FAILED: rms_norm.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/rms_norm.cu -o rms_norm.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/rms_norm.cu".
[3/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_matrix.cu -o q_matrix.cuda.o 
FAILED: q_matrix.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_matrix.cu -o q_matrix.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_matrix.cu".
[4/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/rope.cu -o rope.cuda.o 
FAILED: rope.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/rope.cu -o rope.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/rope.cu".
[5/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/cache.cu -o cache.cuda.o 
FAILED: cache.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/cache.cu -o cache.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/cache.cu".
[6/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/quantize.cu -o quantize.cuda.o 
FAILED: quantize.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/quantize.cu -o quantize.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/quantize.cu".
[7/15] c++ -MMD -MF sampling.o.d -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cpp/sampling.cpp -o sampling.o 
[8/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/lora.cu -o lora.cuda.o 
FAILED: lora.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/lora.cu -o lora.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/lora.cu".
[9/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_attn.cu -o q_attn.cuda.o 
FAILED: q_attn.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_attn.cu -o q_attn.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_attn.cu".
[10/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/h_gemm.cu -o h_gemm.cuda.o 
FAILED: h_gemm.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/h_gemm.cu -o h_gemm.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/h_gemm.cu".
[11/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_gemm.cu -o q_gemm.cuda.o 
FAILED: q_gemm.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_gemm.cu -o q_gemm.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_gemm.cu".
[12/15] /opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_mlp.cu -o q_mlp.cuda.o 
FAILED: q_mlp.cuda.o 
/opt/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -lineinfo -O3 -std=c++17 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_mlp.cu -o q_mlp.cuda.o 
/usr/include/bits/floatn.h(86): error: invalid combination of type specifiers
  typedef __float128 _Float128;
                     ^

/usr/include/bits/floatn-common.h(214): error: invalid combination of type specifiers
  typedef float _Float32;
                ^

/usr/include/bits/floatn-common.h(251): error: invalid combination of type specifiers
  typedef double _Float64;
                 ^

/usr/include/bits/floatn-common.h(268): error: invalid combination of type specifiers
  typedef double _Float32x;
                 ^

/usr/include/bits/floatn-common.h(285): error: invalid combination of type specifiers
  typedef long double _Float64x;
                      ^

5 errors detected in the compilation of "/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cuda/q_mlp.cu".
[13/15] c++ -MMD -MF quantize_func.o.d -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/cpp/quantize_func.cpp -o quantize_func.o 
[14/15] c++ -MMD -MF ext.o.d -DTORCH_EXTENSION_NAME=exllamav2_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/TH -isystem /home/seb/.venvs/x/lib/python3.11/site-packages/torch/include/THC -isystem /opt/cuda/include -isystem /usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -c /home/seb/.venvs/x/lib/python3.11/site-packages/exllamav2/exllamav2_ext/ext.cpp -o ext.o 
ninja: build stopped: subcommand failed.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

Microsoft broke compatibility at one point, and there’s really nothing I can do about it from my end. You’ll want to either update CUDA or downgrade Visual Studio.