bitsandbytes: NameError: name 'str2optimizer8bit_blockwise' is not defined

i got this error when running dreambooth training from here

chiron    | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('https'), PosixPath('//github.com/pypa/get-pip/raw/5eaac1050023df1f5c98b173b248c260023f2278/public/get-pip.py')}
chiron    |   warn(
chiron    | CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
chiron    | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
chiron    |   warn(
chiron    | WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
chiron    | CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
chiron    | /usr/local/lib/python3.9/site-packages/bitsandbytes/cextension.py:48: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
chiron    |   warn(
Steps:   0%|          | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]Traceback (most recent call last):
chiron    |   File "/app/train_dreambooth.py", line 592, in <module>
chiron    |     main()
chiron    |   File "/app/train_dreambooth.py", line 560, in main
chiron    |     optimizer.step()
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/optimizer.py", line 140, in step
chiron    |     self.optimizer.step(closure)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
chiron    |     return wrapped(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/optim/optimizer.py", line 113, in wrapper
chiron    |     return func(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron    |     return func(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 265, in step
chiron    |     self.update_step(group, p, gindex, pindex)
chiron    |   File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron    |     return func(*args, **kwargs)
chiron    |   File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 506, in update_step
chiron    |     F.optimizer_update_8bit_blockwise(
chiron    |   File "/usr/local/lib/python3.9/site-packages/bitsandbytes/functional.py", line 858, in optimizer_update_8bit_blockwise
chiron    |     str2optimizer8bit_blockwise[optimizer_name][0](
chiron    | NameError: name 'str2optimizer8bit_blockwise' is not defined
Steps:   0%|          | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]
chiron    | Traceback (most recent call last):
chiron    |   File "/usr/local/bin/accelerate", line 8, in <module>
chiron    |     sys.exit(main())
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main
chiron    |     args.func(args)
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 837, in launch_command
chiron    |     simple_launcher(args)
chiron    |   File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 354, in simple_launcher
chiron    |     raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

result from python -m bitsandbytes

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = False
COMPUTE_CAPABILITIES_PER_GPU = ['7.0']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable

name 'str2optimizer32bit' is not defined

run inside docker on VM linux ubuntu 20, 64GB, Intel® Xeon® CPU @ 2.20GHz, Nvidia GV100GL [Tesla V100 SXM2 16GB]

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 15
  • Comments: 17 (1 by maintainers)

Most upvoted comments

i was able to solve this by using conda to install the required dependencies:

conda install -c "nvidia/label/cuda-11.7.1" \
	cuda-libraries=11.7.1 \
	cuda-libraries-dev=11.7.1 \
	cudnn

my build is failing here too, with this error. - WSL2, cuda version V10.1.243

tracing the error, it’s failing on the lib.cadam32bit_g32 line of cextension.py (Attribute error) - lib is not None and torch.cuda.is_available, so it doesn’t reach the generate_instructions call, but my COMPILED_WITH_CUDA gets set to False

the problem is that the evaluate_cuda_setup isn’t working correctly, and my binary gets set to libbitsandbytes_cpu.so, even though I have cuda (including libcudart.do).

Looks like this is because i have cuda at usr/lib/wsl/lib, but my cudart_path gets set to /usr/local/cuda/lib64

setting LD_LIBRARY_PATH=“/usr/lib/wsl/lib/” fixed this for me https://github.com/rapidsai/cudf/issues/10308#issuecomment-1042388289

I

@TimDettmers Does the library need to be named literally “cudart.so”? Can it be named “libcudart.so.11.0”?

I am getting

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!

I have the library installed in /usr/lib/x86_64-linux-gnu/libcudart.so.11.0. I tried setting export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/libcudart.so.11.0 but that didn’t work

edit:

okay I think I got it working if I do export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/

I set the LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ and it still not works, is your app run on docker?

Had this problem, upgrading to 0.36post2 solved the issue for me

The fix above solved the issue for me, adjusted for Arch Linux:

LD_LIBRARY_PATH="/opt/cuda/targets/x86_64-linux/lib:$LD_LIBRARY_PATH"