llama.cpp: nvcc fatal : 'f16c': expected a number

Prerequisites

Please answer the following questions for yourself before submitting an issue.

  • [NO] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
  • [YES] I carefully followed the README.md.
  • [YES] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
  • [YES] I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior & Failure Information & Failure Logs

I have successfully compiled the code in CPU-only mode. However, when I attempt to compile it with the ‘-DLLAMA_CUBLAS=1’ flag to enable cuBLAS support, I encounter an error. I expect the compilation to be successful with cuBLAS enabled. Below, I have included the error logs for reference:

#cmake --build . --config Release -- VERBOSE=1
/home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -S/home/sunyuhan/syh/sunyuhan/zju/llama.cpp -B/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build --check-build-system CMakeFiles/Makefile.cmake 0
/home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -E cmake_progress_start /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build/CMakeFiles /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build//CMakeFiles/progress.marks
/usr/bin/make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
/usr/bin/make  -f CMakeFiles/BUILD_INFO.dir/build.make CMakeFiles/BUILD_INFO.dir/depend
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
cd /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build && /home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build/CMakeFiles/BUILD_INFO.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
/usr/bin/make  -f CMakeFiles/BUILD_INFO.dir/build.make CMakeFiles/BUILD_INFO.dir/build
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
make[2]: Nothing to be done for 'CMakeFiles/BUILD_INFO.dir/build'.
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
[  2%] Built target BUILD_INFO
/usr/bin/make  -f CMakeFiles/ggml.dir/build.make CMakeFiles/ggml.dir/depend
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
cd /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build && /home/sunyuhan/syh/sunyuhan/cmake-3.26.3-linux-x86_64/bin/cmake -E cmake_depends "Unix Makefiles" /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build/CMakeFiles/ggml.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
/usr/bin/make  -f CMakeFiles/ggml.dir/build.make CMakeFiles/ggml.dir/build
make[2]: Entering directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
[  5%] Building C object CMakeFiles/ggml.dir/ggml.c.o
/usr/bin/gcc -DGGML_USE_CUBLAS -I/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/. -isystem /usr/local/cuda/include -O3 -DNDEBUG -std=gnu11 -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -mf16c -mfma -mavx -mavx2 -pthread -MD -MT CMakeFiles/ggml.dir/ggml.c.o -MF CMakeFiles/ggml.dir/ggml.c.o.d -o CMakeFiles/ggml.dir/ggml.c.o -c /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/ggml.c
[  8%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
/usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -DGGML_USE_CUBLAS --options-file CMakeFiles/ggml.dir/includes_CUDA.rsp -O3 -DNDEBUG -std=c++11 -mf16c -mfma -mavx -mavx2 -Xcompiler -pthread -MD -MT CMakeFiles/ggml.dir/ggml-cuda.cu.o -MF CMakeFiles/ggml.dir/ggml-cuda.cu.o.d -x cu -c /home/sunyuhan/syh/sunyuhan/zju/llama.cpp/ggml-cuda.cu -o CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'f16c': expected a number
CMakeFiles/ggml.dir/build.make:90: recipe for target 'CMakeFiles/ggml.dir/ggml-cuda.cu.o' failed
make[2]: *** [CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[2]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
CMakeFiles/Makefile2:359: recipe for target 'CMakeFiles/ggml.dir/all' failed
make[1]: *** [CMakeFiles/ggml.dir/all] Error 2
make[1]: Leaving directory '/home/sunyuhan/syh/sunyuhan/zju/llama.cpp/build'
Makefile:100: recipe for target 'all' failed
make: *** [all] Error 2

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

  • Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              48
On-line CPU(s) list: 0-47
Thread(s) per core:  2
Core(s) per socket:  12
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 6146 CPU @ 3.20GHz
Stepping:            4
CPU MHz:             3202.505
CPU max MHz:         3200.0000
CPU min MHz:         1200.0000
BogoMIPS:            6400.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            25344K
NUMA node0 CPU(s):   0-11,24-35
NUMA node1 CPU(s):   12-23,36-47
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm arat pln pts pku ospke md_clear flush_l1d
  • Operating System, e.g. for Linux:

$ uname -a

Linux 7acf3cf70158 5.4.0-91-generic #102~18.04.1-Ubuntu SMP Thu Nov 11 14:46:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • SDK version, e.g. for Linux:
$ make --version

GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ g++ --version

g++ (Ubuntu 8.4.0-1ubuntu1~18.04) 8.4.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

  1. git clone the latest code
  2. mkdir build
  3. cmake … -DLLAMA_CUBLAS=1
  4. cmake --build . --config Release – VERBOSE=1

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 1
  • Comments: 22 (2 by maintainers)

Most upvoted comments

For me, I had 2 nvcc compilers and it was using the “older” one by default. I resolve the problem by specifying the path to the newer one:

cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.2/bin/nvcc
+8
ggerganov on Jul 5, 2023

cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DCMAKE_C_FLAGS='-march=native -pthread' -DCMAKE_CXX_FLAGS='-march=native -pthread' -DLLAMA_AVX=OFF -DLLAMA_AVX2=OFF

the above command works for me.

cuda version V11.0.221

+3
EagleChen on May 26, 2023

Hi @SlyEcho , I tried -DCMAKE_CUDA_ARCHITECTURES=all, another error occured:

CMake Error at /usr/local/share/cmake-3.20/Modules/CMakeDetermineCompilerId.cmake:676 (message):
  Compiling the CUDA compiler identification source file
  "CMakeCUDACompilerId.cu" failed.

  Compiler: /usr/local/cuda-11.0/bin/nvcc

  Build flags:

  Id flags: --keep;--keep-dir;tmp;-gencode=arch=compute_,code=sm_ -v

  

  The output was:

  1

  nvcc fatal : Unsupported gpu architecture 'compute_'

Then I tried -DCMAKE_CUDA_ARCHITECTURES=80, now I got the exactly same error as @sunyuhan19981208 :

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'f16c': expected a number
make[2]: *** [CMakeFiles/ggml.dir/build.make:90: CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

Then I used -DLLAMA_F16C=OFF, this time I got another error:

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'fma': expected a number
make[2]: *** [CMakeFiles/ggml.dir/build.make:90: CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

Then I add -DLLAMA_FMA=OFF, now:

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/data/playground/ll2/llama.cpp/ggml.c: In function ‘ggml_vec_dot_q4_0_q8_0’:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/immintrin.h:97,
                 from /data/playground/ll2/llama.cpp/ggml.c:189:
/opt/rh/devtoolset-8/root/usr/lib/gcc/x86_64-redhat-linux/8/include/fmaintrin.h:63:1: error: inlining failed in call to always_inline ‘_mm256_fmadd_ps’: target specific option mismatch
 _mm256_fmadd_ps (__m256 __A, __m256 __B, __m256 __C)
 ^~~~~~~~~~~~~~~
/data/playground/ll2/llama.cpp/ggml.c:2166:15: note: called from here
         acc = _mm256_fmadd_ps( d, q, acc );
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/ggml.dir/build.make:76: CMakeFiles/ggml.dir/ggml.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

Many thanks for your help!

+2
kexul on May 15, 2023

@kexul You just missed several more flags that are required for legacy Intel CPUs:

Thanks for your detailed information, but none of these worked here sob cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DLLAMA_NATIVE=ON produce:

[  2%] Building C object CMakeFiles/ggml.dir/ggml.c.o
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'arch=native': expected a number
make[2]: *** [CMakeFiles/ggml.dir/build.make:90: CMakeFiles/ggml.dir/ggml-cuda.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:646: CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:101: all] Error 2

After adding -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake, It becomes: cmake .. -DLLAMA_CUBLAS=1 -DCMAKE_CUDA_ARCHITECTURES=80 -DLLAMA_F16C=OFF -DLLAMA_FMA=OFF -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake, now cmake failed.

@EagleChen is right, and generalizing his build command for legacy Intel CPUs:

$ cmake -DCMAKE_CUDA_ARCHITECTURES="all" -DLLAMA_F16C=0 -DLLAMA_FMA=0 -DLLAMA_AVX=0 -DLLAMA_AVX2=0 -DCMAKE_C_FLAGS="-march=native" -DLLAMA_CUBLAS=1 ..

For modern AMD CPUs, it is sufficient to use the build command given by @ggerganov:

cmake -DCMAKE_CUDA_ARCHITECTURES="all" -DCMAKE_C_FLAGS="-march=native" -DLLAMA_CUBLAS=1 ..

However, in the latter case, the default -mf16c, -mfma, … flags are supplied to nvcc instead of cc on some platforms:

...
[  5%] Building CUDA object CMakeFiles/ggml.dir/ggml-cuda.cu.o
nvcc fatal   : 'fma': expected a number
...

In that case, use the build command listed above:

$ cmake -DCMAKE_CUDA_ARCHITECTURES="all" -DLLAMA_F16C=0 -DLLAMA_FMA=0 -DLLAMA_AVX=0 -DLLAMA_AVX2=0 -DCMAKE_C_FLAGS="-march=native" -DLLAMA_CUBLAS=1 ..
+1
luav on Jul 11, 2023

@SlyEcho , Thanks for your continued help! I just tried compile llama.cpp in a docker container with cuda11.8, it works now!

+1
kexul on May 17, 2023

I think these kinds of errors come when the compiler doesn’t know what CPU it is building for, if you want to use AVX and AVX2, you can try the option DLLAMA_NATIVE=ON, if that doesn’t work, add the flag manually with -DCMAKE_C_FLAGS=-march=skylake -CMAKE_CXX_FLAGS=-march=skylake.

Take a look at what options your compiler has: gcc --target-help

+1
SlyEcho on May 15, 2023

I got similar error nvcc fatal : Value 'native' is not defined for option 'gpu-architecture' with cuda 11.0. But I’m not able to upgrade my cuda…

+1
kexul on May 15, 2023

After I change my docker from CUDA11.0 to CUDA11.1, nothing happen. So the lowest version that can run llama.cpp is CUDA11.1?

+1
sunyuhan19981208 on May 15, 2023
Read more comments on GitHub
← llama.cpp: New kv_cache API insufficient to restore model state
llama.cpp: nvcc fatal : Value 'native' is not defined for option 'gpu-architecture' when compiling cuBLAS →