unsloth: Llama3_1_(3B)_GRPO_LoRA vllm error
I was running the notebook https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Advanced_Llama3_1_(3B)_GRPO_LoRA.ipynb But after running the first two cells with the installation commands, when I load the model it shows this error.
[/usr/local/lib/python3.11/dist-packages/unsloth_zoo/vllm_utils.py](https://localhost:8080/#) in patch_vllm_lora_tokenizer()
189 vllm.transformers_utils.tokenizer.get_lora_tokenizer_async = _return_nothing
190
--> 191 import vllm.transformers_utils.tokenizer_group.tokenizer_group
192 vllm.transformers_utils.tokenizer_group.tokenizer_group.get_lora_tokenizer = _return_nothing
193 vllm.transformers_utils.tokenizer_group.tokenizer_group.get_lora_tokenizer_async = _return_nothing
ModuleNotFoundError: No module named 'vllm.transformers_utils.tokenizer_group.tokenizer_group'; 'vllm.transformers_utils.tokenizer_group' is not a package
Any ways to resolve this?
Thank you for your time, Piyush
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (5 by maintainers)
@shimmyshimmer following this old thread: https://github.com/unslothai/unsloth/issues/2104, pip install vllm==0.7.3 seems to work for me for now.
Unsloth now works with all versions of vllm and the latest version! Please update unsloth!!
CC: @catastropiyush @hailong18102002 @jackswl (you already know) @nanowell
@jackswl we’re working on new vllm integration right now which will use the lates version hopefully out by next week. will updateyou!
seems like it works on v0 of vllm but not v1 of vllm
@jackswl yes vllm==0.7.3 works thanks everyone @shimmyshimmer @hailong18102002
Apparrently vllm==0.8.2 works