unsloth: [BUG] RuntimeError: Can't pickle local object 'patch_vllm_compute_dtype..BitsAndBytesConfig'
I’m trying to load and run inference on my GRPO finetuned phi-3.5-mini with VLLM - however this error appears as I load the model with fast_inference = True
from unsloth import FastLanguageModel, is_bfloat16_supported
import torch
max_seq_length = 5000
lora_rank = 16
roundup_model, roundup_tokenizer = FastLanguageModel.from_pretrained(
model_name = "/content/drive/Shareddrives/FYP 2024-2025/Phase-2/SummaryGen/models/phi-3.5-grpo-1-epoch",
max_seq_length = max_seq_length,
dtype = None,
load_in_4bit = True,
fast_inference = True,
max_lora_rank = lora_rank,
gpu_memory_utilization = 0.3,
)
==((====))== Unsloth 2025.3.19: Fast Llama patching. Transformers: 4.50.3. vLLM: 0.8.3. \ /| NVIDIA L4. Num GPUs = 1. Max memory: 22.161 GB. Platform: Linux. O^O/ _/ \ Torch: 2.6.0+cu124. CUDA: 8.9. CUDA Toolkit: 12.4. Triton: 3.2.0 \ / Bfloat16 = TRUE. FA [Xformers = 0.0.29.post3. FA2 = False] “-____-” Free license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored! Unsloth: Your GPU cannot handle sequence lengths of 5000 due to limited GPU memory. Unsloth: Your GPU can only handle approximately the maximum sequence length of 5000. Unsloth: vLLM loading unsloth/phi-3.5-mini-instruct-bnb-4bit with actual GPU utilization = 17.5% Unsloth: Your GPU has CUDA compute capability 8.9 with VRAM = 22.16 GB. Unsloth: Using conservativeness = 1.0. Chunked prefill tokens = 3328. Num Sequences = 128. Unsloth: vLLM’s KV Cache can use up to 1.35 GB. Also swap space = 5 GB. INFO 04-06 13:29:24 [config.py:600] This model supports multiple tasks: {‘score’, ‘classify’, ‘reward’, ‘generate’, ‘embed’}. Defaulting to ‘generate’. INFO 04-06 13:29:24 [config.py:1780] Chunked prefill is enabled with max_num_batched_tokens=3328. WARNING 04-06 13:29:24 [config.py:2468] LoRA with chunked prefill is still experimental and may be unstable. Unsloth: vLLM Bitsandbytes config using kwargs = {‘load_in_8bit’: False, ‘load_in_4bit’: True, ‘bnb_4bit_compute_dtype’: ‘bfloat16’, ‘bnb_4bit_quant_storage’: ‘uint8’, ‘bnb_4bit_quant_type’: ‘nf4’, ‘bnb_4bit_use_double_quant’: True, ‘llm_int8_enable_fp32_cpu_offload’: False, ‘llm_int8_has_fp16_weight’: False, ‘llm_int8_skip_modules’: [], ‘llm_int8_threshold’: 6.0} Unsloth: vLLM Bitsandbytes config using kwargs = {‘load_in_8bit’: False, ‘load_in_4bit’: True, ‘bnb_4bit_compute_dtype’: ‘bfloat16’, ‘bnb_4bit_quant_storage’: ‘uint8’, ‘bnb_4bit_quant_type’: ‘nf4’, ‘bnb_4bit_use_double_quant’: True, ‘llm_int8_enable_fp32_cpu_offload’: False, ‘llm_int8_has_fp16_weight’: False, ‘llm_int8_skip_modules’: [], ‘llm_int8_threshold’: 6.0}
AttributeError Traceback (most recent call last) /usr/local/lib/python3.11/dist-packages/unsloth_zoo/vllm_utils.py in load_vllm(model_name, config, gpu_memory_utilization, max_seq_length, dtype, training, float8_kv_cache, random_state, enable_lora, max_lora_rank, max_loras, use_async, use_engine, disable_log_stats, enforce_eager, enable_prefix_caching, compilation_config, conservativeness, max_logprobs, use_bitsandbytes) 1027 else: -> 1028 llm = LLM(**engine_args) 1029 pass
21 frames AttributeError: Can’t pickle local object ‘patch_vllm_compute_dtype.<locals>.BitsAndBytesConfig’
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last) /usr/local/lib/python3.11/dist-packages/unsloth_zoo/vllm_utils.py in load_vllm(model_name, config, gpu_memory_utilization, max_seq_length, dtype, training, float8_kv_cache, random_state, enable_lora, max_lora_rank, max_loras, use_async, use_engine, disable_log_stats, enforce_eager, enable_prefix_caching, compilation_config, conservativeness, max_logprobs, use_bitsandbytes) 1049 ) 1050 else: -> 1051 raise RuntimeError(error) 1052 pass 1053 pass
RuntimeError: Can’t pickle local object ‘patch_vllm_compute_dtype.<locals>.BitsAndBytesConfig’
This is my installation setup:
%%capture
!pip install --no-deps unsloth vllm
# [NOTE] Do the below ONLY in Colab! Use [[pip install unsloth vllm]]
# Skip restarting message in Colab
import sys, re, requests; modules = list(sys.modules.keys())
for x in modules: sys.modules.pop(x) if "PIL" in x or "google" in x else None
!pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl==0.15.2 triton cut_cross_entropy unsloth_zoo
!pip install sentencepiece protobuf datasets huggingface_hub hf_transfer
# vLLM requirements - vLLM breaks Colab due to reinstalling numpy
f = requests.get("https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/requirements/common.txt").content
with open("vllm_requirements.txt", "wb") as file:
file.write(re.sub(rb"(transformers|numpy|xformers)[^\n]{1,}\n", b"", f))
!pip install -r vllm_requirements.txt
When I try to run inference without VLLM, the model does not, for some reason, perform as expected. Is there any fix/workaround to this? Is it a version compatibility issue? I had faced “VLLM Server Connection refused” when fine-tuning my model with GRPO and downgrading trl to v0.15.2 helped - inference worked as well - so I used the same installation setup. Please help!
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 15
vllm was updated into v0.8.3 yesterday. Downgrade into ==0.8.2 might help.
The following requirements.txt file seems to work at the moment: (06.05.2025 - May '25)
(those are the latest versions of unsloth and triton - that get installed automatically when installing
pip3 install unsloth vllm. Also by default this would install vllm==0.8.5.post1, however, that is breaking).For reference here are also the other important libraries and their versions:
Also, this was installed using
pipin a conda enviornment with python=3.11.I think I figured it out had to do the following on a vast GPU host.