sglang: [Bug] liuhaotian/llava-v1.6-mistral-7b doesn't load
When trying to load the Mistral variant of LLaVa 1.6, I get an expected error:
python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-mistral-7b --chat-template vicuna_v1.1 --port 30000
ValueError: The checkpoint you are trying to load has model type `llava_mistral` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date
Transformers doesn’t treat the LLaVa variants any differently, they all use the same config. I think this could be easily fixed by adding a mapping from llava_mistral to the LlavaConfig in the config mapping.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 8
- Comments: 16 (6 by maintainers)
I went and patched the 1.6 Mistral model to work more smoothly with SGLang: SurfaceData/llava-v1.6-mistral-7b-sglang. I also tweaked the vicuna version as well SurfaceData/llava-v1.6-vicuna-7b-sglang
This way “kind” of work:
You can hack
hf_transformers_utils.pyin SGLang to make it work, but I’m not sure if the later loaded model is functional correct. Still, I don’t think this problem should be fixed in SGLang.Another way is:
git clone https://huggingface.co/liuhaotian/llava-v1.6-mistral-7bllava-v1.6-mistral-7b/config.jsonby changing its architecture toLlavaLlamaForCausalLM.--model-path ./llava-v1.6-mistral-7bI would recommend this way as Mistral has exactly the same architecture as Llama2 so the checkpoints are sharable.
Same issue here.
ValueError: The checkpoint you are trying to load has model type " llava_mistral " but Transformers does not recognize this architecture.for liuhaotian/llava-v1.6-mistral-7bThanks @dillonalaird , when trying to run your version of llava-1.6 using
I ran into error
RuntimeError: BatchPrefillWithPagedKVCache failed to dispatch with dtype HalfHow are you running this for it to work for you? thanks!
Thanks @comaniac this gets me past the
you are trying to load has model typellava_mistral`` but the next blocker isllava-v1.6-mistral-7b does not appear to have a file named preprocessor_config.json