ArcticInference: Worker process getting killed

I’m trying to run VLLM using the command:

VLLM_LOGGING_LEVEL=DEBUG CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=1,2,3,4 vllm serve \
    meta-llama/Llama-3.1-8B-Instruct \
    --quantization None \
    --port 9434 \
    --tensor-parallel-size 4 \
    --speculative-config '{
        "method": "arctic",
        "model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": true
    }'

Getting this error below, this is all that is printed in the logs, there is no stack trace.

(VllmWorker rank=0 pid=2480291) INFO 05-07 19:18:17 [loader.py:458] Loading weights took 3.07 seconds
(VllmWorker rank=0 pid=2480291) Process SpawnProcess-1:1:
CRITICAL 05-07 19:18:17 [ulysses.py:230] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
CRITICAL 05-07 19:18:17 [core_client.py:359] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
(VllmWorker rank=0 pid=2480291) Traceback (most recent call last):
Killed

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 30 (5 by maintainers)

Most upvoted comments

@sfc-gh-yewang Thank you for the explanation! That helps a lot. I also just found that VLLM recently realized this issue and merged a fix at vllm-project/vllm#17929

@sfc-gh-yewang I increase the max_model_len to 64k tokens and added “disable_by_batch_size”:32 in speculative_config, working very well now! Was averaging around 250 tokens/s yesterday. I did get the error from the kv cache “assert num_new_blocks > 0”, but I believe this was just from setting gpu_memory_utilization too high. Will do further testing, but closing this issue.

@Xiuyu-Li spec decoding should still be applied with temperature=0. We will see if we can reproduce and fix the hang with temp > 0.