sglang: [Bug] liuhaotian/llava-v1.6-mistral-7b doesn't load

When trying to load the Mistral variant of LLaVa 1.6, I get an expected error:

python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-mistral-7b --chat-template vicuna_v1.1 --port 30000
ValueError: The checkpoint you are trying to load has model type `llava_mistral` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date

Transformers doesn’t treat the LLaVa variants any differently, they all use the same config. I think this could be easily fixed by adding a mapping from llava_mistral to the LlavaConfig in the config mapping.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 8
  • Comments: 16 (6 by maintainers)

Most upvoted comments

I went and patched the 1.6 Mistral model to work more smoothly with SGLang: SurfaceData/llava-v1.6-mistral-7b-sglang. I also tweaked the vicuna version as well SurfaceData/llava-v1.6-vicuna-7b-sglang

This way “kind” of work:

>>> from transformers.models.llava.configuration_llava import LlavaConfig
>>> LlavaConfig.from_pretrained("liuhaotian/llava-v1.6-mistral-7b")
You are using a model of type llava_mistral to instantiate a model of type llava. This is not supported for all configurations of models and can yield errors.
LlavaConfig {
  "_name_or_path": "mistralai/Mistral-7B-Instruct-v0.2",
  "architectures": [
    "LlavaMistralForCausalLM"
  ],
...

You can hack hf_transformers_utils.py in SGLang to make it work, but I’m not sure if the later loaded model is functional correct. Still, I don’t think this problem should be fixed in SGLang.

Another way is:

  1. Download the model to local disk: git clone https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b
  2. Modify llava-v1.6-mistral-7b/config.json by changing its architecture to LlavaLlamaForCausalLM.
  3. Load the modified checkpoint with --model-path ./llava-v1.6-mistral-7b

I would recommend this way as Mistral has exactly the same architecture as Llama2 so the checkpoints are sharable.

Same issue here. ValueError: The checkpoint you are trying to load has model type " llava_mistral " but Transformers does not recognize this architecture. for liuhaotian/llava-v1.6-mistral-7b

I’ve created a repo for llava-v1.6-34B that works https://huggingface.co/dillonlaird/hf-llava-v1.6-34b looking at the other llava-hf models you basically need to add the preprocessor_config.json, which should be the same, and then add the added_tokens.json file

Thanks @dillonalaird , when trying to run your version of llava-1.6 using

python3 -m sglang.launch_server \
--model-path dillonlaird/hf-llava-v1.6-34b \
--tokenizer-path dillonlaird/hf-llava-v1.6-34b \
--host "0.0.0.0" \
--tp 2 \
--port 2222 \
--model-mode flashinfer

I ran into error RuntimeError: BatchPrefillWithPagedKVCache failed to dispatch with dtype Half

How are you running this for it to work for you? thanks!

Thanks @comaniac this gets me past the you are trying to load has model type llava_mistral`` but the next blocker is llava-v1.6-mistral-7b does not appear to have a file named preprocessor_config.json