bitsandbytes: NameError: name 'str2optimizer8bit_blockwise' is not defined
i got this error when running dreambooth training from here
chiron | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('https'), PosixPath('//github.com/pypa/get-pip/raw/5eaac1050023df1f5c98b173b248c260023f2278/public/get-pip.py')}
chiron | warn(
chiron | CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
chiron | /usr/local/lib/python3.9/site-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
chiron | warn(
chiron | WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
chiron | CUDA SETUP: Loading binary /usr/local/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so...
chiron | /usr/local/lib/python3.9/site-packages/bitsandbytes/cextension.py:48: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
chiron | warn(
Steps: 0%| | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]Traceback (most recent call last):
chiron | File "/app/train_dreambooth.py", line 592, in <module>
chiron | main()
chiron | File "/app/train_dreambooth.py", line 560, in main
chiron | optimizer.step()
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/optimizer.py", line 140, in step
chiron | self.optimizer.step(closure)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 65, in wrapper
chiron | return wrapped(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/optim/optimizer.py", line 113, in wrapper
chiron | return func(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron | return func(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 265, in step
chiron | self.update_step(group, p, gindex, pindex)
chiron | File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
chiron | return func(*args, **kwargs)
chiron | File "/usr/local/lib/python3.9/site-packages/bitsandbytes/optim/optimizer.py", line 506, in update_step
chiron | F.optimizer_update_8bit_blockwise(
chiron | File "/usr/local/lib/python3.9/site-packages/bitsandbytes/functional.py", line 858, in optimizer_update_8bit_blockwise
chiron | str2optimizer8bit_blockwise[optimizer_name][0](
chiron | NameError: name 'str2optimizer8bit_blockwise' is not defined
Steps: 0%| | 0/800 [00:01<?, ?it/s, loss=0.183, lr=5e-6]
chiron | Traceback (most recent call last):
chiron | File "/usr/local/bin/accelerate", line 8, in <module>
chiron | sys.exit(main())
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main
chiron | args.func(args)
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 837, in launch_command
chiron | simple_launcher(args)
chiron | File "/usr/local/lib/python3.9/site-packages/accelerate/commands/launch.py", line 354, in simple_launcher
chiron | raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
result from python -m bitsandbytes
++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = False
COMPUTE_CAPABILITIES_PER_GPU = ['7.0']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Running a quick check that:
+ library is importable
+ CUDA function is callable
name 'str2optimizer32bit' is not defined
run inside docker on VM linux ubuntu 20, 64GB, Intel® Xeon® CPU @ 2.20GHz, Nvidia GV100GL [Tesla V100 SXM2 16GB]
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 17
- Comments: 17
i was able to solve this by using conda to install the required dependencies:
my build is failing here too, with this error. - WSL2, cuda version V10.1.243
tracing the error, it’s failing on the
lib.cadam32bit_g32line of cextension.py (Attribute error) - lib is not None and torch.cuda.is_available, so it doesn’t reach the generate_instructions call, but my COMPILED_WITH_CUDA gets set to Falsethe problem is that the evaluate_cuda_setup isn’t working correctly, and my binary gets set to
libbitsandbytes_cpu.so, even though I have cuda (including libcudart.do).Looks like this is because i have cuda at
usr/lib/wsl/lib, but my cudart_path gets set to/usr/local/cuda/lib64setting LD_LIBRARY_PATH=“/usr/lib/wsl/lib/” fixed this for me https://github.com/rapidsai/cudf/issues/10308#issuecomment-1042388289
I
I set the LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ and it still not works, is your app run on docker?
The fix above solved the issue for me, adjusted for Arch Linux:
LD_LIBRARY_PATH="/opt/cuda/targets/x86_64-linux/lib:$LD_LIBRARY_PATH"Had this problem, upgrading to 0.36post2 solved the issue for me