ArcticInference: Worker process getting killed
I’m trying to run VLLM using the command:
VLLM_LOGGING_LEVEL=DEBUG CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1,2,3,4 vllm serve \
meta-llama/Llama-3.1-8B-Instruct \
--quantization None \
--port 9434 \
--tensor-parallel-size 4 \
--speculative-config '{
"method": "arctic",
"model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
"num_speculative_tokens": 3,
"enable_suffix_decoding": true
}'
Getting this error below, this is all that is printed in the logs, there is no stack trace.
(VllmWorker rank=0 pid=2480291) INFO 05-07 19:18:17 [loader.py:458] Loading weights took 3.07 seconds
(VllmWorker rank=0 pid=2480291) Process SpawnProcess-1:1:
CRITICAL 05-07 19:18:17 [ulysses.py:230] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
CRITICAL 05-07 19:18:17 [core_client.py:359] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
(VllmWorker rank=0 pid=2480291) Traceback (most recent call last):
Killed
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 30 (5 by maintainers)
@sfc-gh-yewang Thank you for the explanation! That helps a lot. I also just found that VLLM recently realized this issue and merged a fix at vllm-project/vllm#17929
@sfc-gh-yewang I increase the max_model_len to 64k tokens and added “disable_by_batch_size”:32 in speculative_config, working very well now! Was averaging around 250 tokens/s yesterday. I did get the error from the kv cache “assert num_new_blocks > 0”, but I believe this was just from setting gpu_memory_utilization too high. Will do further testing, but closing this issue.
@Xiuyu-Li spec decoding should still be applied with temperature=0. We will see if we can reproduce and fix the hang with temp > 0.