CTranslate2: CUDA 12 support (libcublas.so.11 is not found)

The major release 4.0.0 supports CUDA 12 now. Feel free to test it if there is any issue about this. Thanks!

If anyone else is encountering this on Colab (or other Ubuntu jammy systems), one workaround for now is to install the libcublas11 package. This can be accomplished by including the following at the start of a Colab notebook:

!apt install libcublas11

Here is updated ctranslate2 install for Ubuntu 22.04 with cuda 12.x. You should pip uninstall the existing ctranslate2 before proceeding.

intel mkl

wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB | gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list

apt update
apt install intel-oneapi-mkl intel-oneapi-mkl-devel

ubuntu 22.04

apt install git build-essential
apt install cmake
apt install libomp5 libomp-dev
apt install cuda-toolkit
apt install zlib1g
apt install libcudnn8 libcudnn8-dev libcudnn8-samples

clone

git clone --recursive https://github.com/OpenNMT/CTranslate2.git

build

cd CTranslate2
mkdir build 
cd build
cmake -DWITH_MKL=ON -DWITH_CUDA=ON -DWITH_CUDNN=ON  -DOPENMP_RUNTIME=INTEL  ..
make -j8
make install

python binding

cd ../python
pip install -r install_requirements.txt
CTRANSLATE2_ROOT=/usr/local python setup.py bdist_wheel

final step:

pip install the file generated in the “dist” folder. The name of the file is different due to python version and arch.

If you have to use cuda 12 and want a quick hacky fix, one way to try is:

ln -s /usr/local/cuda/lib64/libcublas.so.12 /usr/local/cuda/lib64/libcublas.so.11

since ctranslate2 uses standard cublas apis, this usually won’t produce an error. You can keep all other configs as-is.

The tests shows that “Faster-Whisper with CUDA v12” has -10% drop in performance.

RTX 3050 GPU:

float16:      -10% drop in speed 
bfloat16:      -8% drop in speed
int8_bfloat16:  0% same

build manually possible. bellow is a bash history to reference:

   10  apt install git build-essential
   13  apt install cmake
   21  apt-get install libomp5
   23  apt install libomp-dev
   24  apt-get install linux-headers-$(uname -r)
   25  apt-get install cuda-toolkit
   26  wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
   27  apt-get install wget
   28  wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
   29  dpkg -i cuda-keyring_1.1-1_all.deb
   30  rm cuda-keyring_1.1-1_all.deb
   31  cd ..
   32  cd
   33  apt update
   34  apt-get install cuda-toolkit
   35  wget https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.6/local_installers/12.x/cudnn-local-repo-ubuntu2204-8.9.6.50_1.0-1_amd64.deb/
   36  ls
   37  ls /
   38  ls /root
   39  dpkg -i /root/cudnn-local-repo-ubuntu2204-8.9.6.50_1.0-1_amd64.deb
   40  sudo cp /var/cudnn-local-repo-ubuntu2204-8.9.6.50/cudnn-local-1998375D-keyring.gpg /usr/share/keyrings/
   41  cp /var/cudnn-local-repo-ubuntu2204-8.9.6.50/cudnn-local-1998375D-keyring.gpg /usr/share/keyrings/
   42  dpkg -i /root/cudnn-local-repo-ubuntu2204-8.9.6.50_1.0-1_amd64.deb
   43  apt-get install zlib1g
   44  apt-get update
   45  ls /root
   46  apt-cache search libcudnn
   49  apt-get install libcudnn8 libcudnn8-dev libcudnn8-samples