bitsandbytes: NameError: name 'str2optimizer8bit_blockwise' is not defined
i got this error when running dreambooth training from here
chiron | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('https'), PosixPath('//github.com/pypa/get-pip/raw/5eaac1050023df1f5c98b173b248c260023f2278/public/get-pip.py')}
chiron | warn(
chiron | CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
chiron | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
chiron | warn(
chiron | WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
chiron | CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
chiron | /usr/local/lib/python3.9/site-packages/bitsandbytes/cextension.py:48: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
chiron | warn(
Steps: 0%| | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]Traceback (most recent call last):
chiron | File "/app/train_dreambooth.py", line 592, in <module>
chiron | main()
chiron | File "/app/train_dreambooth.py", line 560, in main
chiron | optimizer.step()
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/optimizer.py", line 140, in step
chiron | self.optimizer.step(closure)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
chiron | return wrapped(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/optim/optimizer.py", line 113, in wrapper
chiron | return func(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron | return func(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 265, in step
chiron | self.update_step(group, p, gindex, pindex)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron | return func(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 506, in update_step
chiron | F.optimizer_update_8bit_blockwise(
chiron | File "/usr/local/lib/python3.9/site-packages/bitsandbytes/functional.py", line 858, in optimizer_update_8bit_blockwise
chiron | str2optimizer8bit_blockwise[optimizer_name][0](
chiron | NameError: name 'str2optimizer8bit_blockwise' is not defined
Steps: 0%| | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]
chiron | Traceback (most recent call last):
chiron | File "/usr/local/bin/accelerate", line 8, in <module>
chiron | sys.exit(main())
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main
chiron | args.func(args)
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 837, in launch_command
chiron | simple_launcher(args)
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 354, in simple_launcher
chiron | raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
result from python -m bitsandbytes
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = False
COMPUTE_CAPABILITIES_PER_GPU = ['7.0']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Running a quick check that:
+ library is importable
+ CUDA function is callable
name 'str2optimizer32bit' is not defined
run inside docker on VM linux ubuntu 20, 64GB, Intel® Xeon® CPU @ 2.20GHz, Nvidia GV100GL [Tesla V100 SXM2 16GB]
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 15
- Comments: 17 (1 by maintainers)
i was able to solve this by using conda to install the required dependencies:
my build is failing here too, with this error. - WSL2, cuda version V10.1.243
tracing the error, it’s failing on the
lib.cadam32bit_g32line of cextension.py (Attribute error) - lib is not None and torch.cuda.is_available, so it doesn’t reach the generate_instructions call, but my COMPILED_WITH_CUDA gets set to Falsethe problem is that the evaluate_cuda_setup isn’t working correctly, and my binary gets set to
libbitsandbytes_cpu.so, even though I have cuda (including libcudart.do).Looks like this is because i have cuda at
usr/lib/wsl/lib, but my cudart_path gets set to/usr/local/cuda/lib64setting LD_LIBRARY_PATH=“/usr/lib/wsl/lib/” fixed this for me https://github.com/rapidsai/cudf/issues/10308#issuecomment-1042388289
I
I set the LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ and it still not works, is your app run on docker?
Had this problem, upgrading to 0.36post2 solved the issue for me
The fix above solved the issue for me, adjusted for Arch Linux:
LD_LIBRARY_PATH="/opt/cuda/targets/x86_64-linux/lib:$LD_LIBRARY_PATH"