LLaVA: [Usage] ImportError: cannot import name 'LlavaLlamaForCausalLM' from 'llava.model'
Describe the issue
Issue: So i updated the repo and now i can not start the server anymore. I deleted the repo and cloned it again, but get the same error.
System: Win10 WSL2 Ubuntu pip 24.0 from /home/tobias/.local/lib/python3.10/site-packages/pip (python 3.10)
Command:
git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
conda create --name LLaVA python=3.10.12
conda activate LLaV
pip install -e .
pip install flash-attn --no-build-isolation
python3 -m llava.serve.controller --host 0.0.0.0 --port 10000
Log:
python3 -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-v1.5-13b --load-4bit
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
__import__(pkg_name)
File "/mnt/a/KI/LLaVA/llava/__init__.py", line 1, in <module>
from .model import LlavaLlamaForCausalLM
ImportError: cannot import name 'LlavaLlamaForCausalLM' from 'llava.model' (/mnt/a/KI/LLaVA/llava/model/__init__.py)
Packages:
pip list
Package Version Editable project location
------------------------- ------------- -------------------------
accelerate 0.21.0
aiofiles 23.2.1
aiohttp 3.9.3
aiosignal 1.3.1
altair 5.2.0
annotated-types 0.6.0
anyio 4.2.0
async-timeout 4.0.3
attrs 23.2.0
bitsandbytes 0.41.0
blinker 1.4
CacheControl 0.12.10
cachy 0.3.0
certifi 2023.11.17
chardet 4.0.0
charset-normalizer 3.3.2
cleo 0.8.1
click 8.1.7
clikit 0.6.2
cmake 3.28.1
colorama 0.4.6
command-not-found 0.3
contourpy 1.2.0
crashtest 0.3.1
cryptography 3.4.8
cycler 0.12.1
dbus-python 1.2.18
distlib 0.3.4
distro 1.7.0
distro-info 1.1+ubuntu0.2
einops 0.6.1
einops-exts 0.0.4
exceptiongroup 1.2.0
fastapi 0.109.0
ffmpy 0.3.1
filelock 3.13.1
flash-attn 2.5.2
fonttools 4.47.2
frozenlist 1.4.1
fsspec 2023.12.2
gradio 4.16.0
gradio_client 0.8.1
h11 0.14.0
html5lib 1.1
httpcore 0.17.3
httplib2 0.20.2
httpx 0.24.0
huggingface-hub 0.20.3
idna 3.6
importlib-metadata 4.6.4
importlib-resources 6.1.1
jeepney 0.7.1
Jinja2 3.1.3
joblib 1.3.2
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
keyring 23.5.0
kiwisolver 1.4.5
launchpadlib 1.10.16
lazr.restfulclient 0.14.4
lazr.uri 1.0.6
linkify-it-py 2.0.2
lit 17.0.6
llava 1.2.2.post1 /mnt/a/KI/LLaVA
lockfile 0.12.2
markdown-it-py 2.2.0
markdown2 2.4.12
MarkupSafe 2.1.4
matplotlib 3.8.2
mdit-py-plugins 0.3.3
mdurl 0.1.2
more-itertools 8.10.0
mpmath 1.3.0
msgpack 1.0.3
multidict 6.0.4
netifaces 0.11.0
networkx 3.2.1
ninja 1.11.1.1
numpy 1.26.3
nvidia-cublas-cu11 11.10.3.66
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu11 8.5.0.96
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu11 10.9.0.58
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu11 10.2.10.91
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu11 11.7.4.91
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu11 2.14.3
nvidia-nccl-cu12 2.18.1
nvidia-nvjitlink-cu12 12.3.101
nvidia-nvtx-cu11 11.7.91
nvidia-nvtx-cu12 12.1.105
oauthlib 3.2.0
orjson 3.9.12
packaging 23.2
pandas 2.2.0
pastel 0.2.1
peft 0.4.0
pexpect 4.8.0
pillow 10.2.0
pip 24.0
pkginfo 1.8.2
platformdirs 2.5.1
poetry-core 1.0.7
psutil 5.9.8
ptyprocess 0.7.0
pydantic 2.6.1
pydantic_core 2.16.2
pydub 0.25.1
Pygments 2.17.2
PyGObject 3.42.1
PyJWT 2.3.0
pylev 1.2.0
pyparsing 2.4.7
python-apt 2.4.0+ubuntu2
python-dateutil 2.8.2
python-multipart 0.0.6
pytz 2023.4
PyYAML 5.4.1
referencing 0.33.0
regex 2023.12.25
requests 2.31.0
requests-toolbelt 0.9.1
rich 13.7.0
rpds-py 0.17.1
ruff 0.2.1
safetensors 0.4.2
scikit-learn 1.2.2
scipy 1.12.0
SecretStorage 3.3.1
semantic-version 2.10.0
sentencepiece 0.1.99
setuptools 59.6.0
shellingham 1.4.0
shortuuid 1.0.11
six 1.16.0
sniffio 1.3.0
starlette 0.35.1
svgwrite 1.4.3
sympy 1.12
systemd-python 234
threadpoolctl 3.2.0
timm 0.6.13
tokenizers 0.15.1
tomlkit 0.12.0
toolz 0.12.1
torch 2.1.2
torchvision 0.16.2
tqdm 4.66.1
transformers 4.37.2
triton 2.1.0
typer 0.9.0
typing_extensions 4.9.0
tzdata 2023.4
ubuntu-advantage-tools 8001
uc-micro-py 1.0.2
ufw 0.36.1
unattended-upgrades 0.1
urllib3 2.2.0
uvicorn 0.27.0.post1
virtualenv 20.13.0+ds
wadllib 1.3.6
wavedrom 2.0.3.post3
webencodings 0.5.1
websockets 11.0.3
wheel 0.37.1
yarl 1.9.4
zipp 1.0.0
Screenshots:
About this issue
- Original URL
- State: closed
- Created 5 months ago
- Comments: 22 (3 by maintainers)
The above order worked for me. The order is important.
For me it was problem with packages like deepspeed etc. that led to above. But any issue will lead to it. This is the combo that worked for me:
In my and the above case the deepspeed error was hit with inference too.
I didn’t check exactly which deepspeed version was required. I just did pip install accelerate deepspeed --upgrade and it started working, so I wanted to remember the versions that worked so wrote those down.
I got that
libcudart.so.12.0: cannot open shared object file: No such file or directorytype errors, but that’s just because you must not have set CUDA_HOME corretly or do not have cuda toolkit installed for (there) cuda 12.1. It shouldn’t be looking for cuda 11 if you really freshly installed with CUDA_HOME and that PIP stuff set.A last thing I may have done is to literally relink the cuda directory:
But make sure all your cuda stuff is consistent.
We find that this is due to flash-attn compiled previously with a different version of pytorch. Please reinstall that with:
Try importing the packages without the “try, except” block for a more informative error.
Probably related to flash attn installation.
In my case, the following worked: