unsloth: Llama3_1_(3B)_GRPO_LoRA vllm error

I was running the notebook https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_1_(3B)_GRPO_LoRA.ipynb But after running the first two cells with the installation commands, when I load the model it shows this error.

[/usr/local/lib/python3.11/dist-packages/unsloth_zoo/vllm_utils.py](https://localhost:8080/#) in patch_vllm_lora_tokenizer()
    189         vllm.transformers_utils.tokenizer.get_lora_tokenizer_async = _return_nothing
    190 
--> 191         import vllm.transformers_utils.tokenizer_group.tokenizer_group
    192         vllm.transformers_utils.tokenizer_group.tokenizer_group.get_lora_tokenizer = _return_nothing
    193         vllm.transformers_utils.tokenizer_group.tokenizer_group.get_lora_tokenizer_async = _return_nothing

ModuleNotFoundError: No module named 'vllm.transformers_utils.tokenizer_group.tokenizer_group'; 'vllm.transformers_utils.tokenizer_group' is not a package

Any ways to resolve this?

Thank you for your time, Piyush

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (5 by maintainers)

Most upvoted comments

@shimmyshimmer following this old thread: https://github.com/unslothai/unsloth/issues/2104, pip install vllm==0.7.3 seems to work for me for now.

Unsloth now works with all versions of vllm and the latest version! Please update unsloth!!

CC: @catastropiyush @hailong18102002 @jackswl (you already know) @nanowell

@jackswl we’re working on new vllm integration right now which will use the lates version hopefully out by next week. will updateyou!

seems like it works on v0 of vllm but not v1 of vllm

@jackswl yes vllm==0.7.3 works thanks everyone @shimmyshimmer @hailong18102002

Apparrently vllm==0.8.2 works