vllm: [Bug]: SIGSEGV received at time=1721904360 on cpu 140, Fatal Python error: Segmentation fault
Your current environment
My environment setup involving two 8xH100 nodes is detailed in https://github.com/vllm-project/vllm/issues/6775; therefore, I will omit it here for brevity.
🐛 Describe the bug
I am trying to run lm-evaluation-harness with vllm as the backend. The evaluation should run either with TP=16 or with TP=8 + PP=2.
The problem I am facing happens after I run this command:
lm_eval \
--model vllm \
--model_args pretrained="meta-llama/Meta-Llama-3.1-8B-Instruct",dtype=auto,gpu_memory_utilization=0.8,add_bos_token=True,max_model_len=2048,tensor_parallel_size=16,distributed_executor_backend="ray" \
--tasks winogrande \
--num_fewshot 5 \
--batch_size auto \
--device cuda
and it looks like this:
INFO 07-25 10:44:52 llm_engine.py:176] Initializing an LLM engine (v0.5.3.post1) with config: model='/home/meta-llama/Meta-Llama-3-8B-Instruct', speculative_config=None, tokenizer='/home/meta-llama/Meta-Llama-3-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=16, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=1234, served_model_name=/home/meta-llama/Meta-Llama-3-8B-Instruct, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 07-25 10:45:26 utils.py:784] Found nccl from library libnccl.so.2
INFO 07-25 10:45:26 pynccl.py:63] vLLM is using nccl==2.20.5
(RayWorkerWrapper pid=23732) INFO 07-25 10:45:26 utils.py:784] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=23732) INFO 07-25 10:45:26 pynccl.py:63] vLLM is using nccl==2.20.5
WARNING 07-25 10:45:39 custom_all_reduce.py:69] Custom allreduce is disabled because this process group spans across nodes.
(RayWorkerWrapper pid=1776, ip=192.168.201.208) WARNING 07-25 10:45:39 custom_all_reduce.py:69] Custom allreduce is disabled because this process group spans across nodes.
(RayWorkerWrapper pid=2844, ip=192.168.201.208) INFO 07-25 10:45:26 utils.py:784] Found nccl from library libnccl.so.2 [repeated 14x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(RayWorkerWrapper pid=2844, ip=192.168.201.208) INFO 07-25 10:45:26 pynccl.py:63] vLLM is using nccl==2.20.5 [repeated 14x across cluster]
INFO 07-25 10:45:39 shm_broadcast.py:241] vLLM message queue communication handle: Handle(connect_ip='192.168.201.210', local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fb11c203550>, local_subscribe_port=38653, local_sync_port=49747, remote_subscribe_port=45329, remote_sync_port=39441)
INFO 07-25 10:45:39 model_runner.py:680] Starting to load model /home/meta-llama/Meta-Llama-3-8B-Instruct...
(RayWorkerWrapper pid=23732) INFO 07-25 10:45:39 model_runner.py:680] Starting to load model /home/meta-llama/Meta-Llama-3-8B-Instruct...
Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:00<00:00, 20.10it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:00<00:00, 17.74it/s]
INFO 07-25 10:45:39 model_runner.py:692] Loading model weights took 0.9685 GB
(RayWorkerWrapper pid=23732) INFO 07-25 10:45:39 model_runner.py:692] Loading model weights took 0.9685 GB
INFO 07-25 10:45:52 distributed_gpu_executor.py:56] # GPU blocks: 168234, # CPU blocks: 16384
INFO 07-25 10:45:54 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 07-25 10:45:54 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(RayWorkerWrapper pid=23732) INFO 07-25 10:45:54 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(RayWorkerWrapper pid=23732) INFO 07-25 10:45:54 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
(RayWorkerWrapper pid=24649) WARNING 07-25 10:45:39 custom_all_reduce.py:69] Custom allreduce is disabled because this process group spans across nodes. [repeated 14x across cluster]
(RayWorkerWrapper pid=2844, ip=192.168.201.208) INFO 07-25 10:45:39 model_runner.py:680] Starting to load model /home/meta-llama/Meta-Llama-3-8B-Instruct... [repeated 14x across cluster]
(RayWorkerWrapper pid=2080, ip=192.168.201.208) INFO 07-25 10:45:40 model_runner.py:692] Loading model weights took 0.9685 GB [repeated 14x across cluster]
(RayWorkerWrapper pid=24036) *** SIGSEGV received at time=1721904360 on cpu 140 ***
(RayWorkerWrapper pid=24036) PC: @ 0x7f17f832d44a (unknown) addProxyOpIfNeeded()
(RayWorkerWrapper pid=24036) @ 0x7f1bc25d8090 36407536 (unknown)
(RayWorkerWrapper pid=24036) @ 0xabaf7f0 80 (unknown)
(RayWorkerWrapper pid=24036) @ 0xabaf7f0 (unknown) (unknown)
(RayWorkerWrapper pid=24036) [2024-07-25 10:46:00,050 E 24036 24036] logging.cc:440: *** SIGSEGV received at time=1721904360 on cpu 140 ***
(RayWorkerWrapper pid=24036) [2024-07-25 10:46:00,050 E 24036 24036] logging.cc:440: PC: @ 0x7f17f832d44a (unknown) addProxyOpIfNeeded()
(RayWorkerWrapper pid=24036) [2024-07-25 10:46:00,052 E 24036 24036] logging.cc:440: @ 0x7f1bc25d8090 36407536 (unknown)
(RayWorkerWrapper pid=24036) [2024-07-25 10:46:00,054 E 24036 24036] logging.cc:440: @ 0xabaf7f0 80 (unknown)
(RayWorkerWrapper pid=24036) [2024-07-25 10:46:00,057 E 24036 24036] logging.cc:440: @ 0xabaf7f0 (unknown) (unknown)
(RayWorkerWrapper pid=24036) Fatal Python error: Segmentation fault
(RayWorkerWrapper pid=24036)
(RayWorkerWrapper pid=24036) Stack (most recent call first):
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 257 in ncclAllReduce
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/pynccl.py", line 118 in all_reduce
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 291 in all_reduce
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/communication_op.py", line 11 in tensor_model_parallel_all_reduce
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/linear.py", line 787 in forward
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541 in _call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 89 in forward
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541 in _call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 255 in forward
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541 in _call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 322 in forward
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541 in _call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/llama.py", line 422 in forward
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541 in _call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532 in _wrapped_call_impl
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1398 in capture
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/model_runner.py", line 1173 in capture_model
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115 in decorate_context
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 236 in _warm_up_model
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py", line 220 in initialize_cache
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 374 in execute_method
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/ray/util/tracing/tracing_helper.py", line 467 in _resume_span
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/ray/_private/function_manager.py", line 691 in actor_method_executor
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/ray/_private/worker.py", line 887 in main_loop
(RayWorkerWrapper pid=24036) File "/usr/local/lib/python3.10/dist-packages/ray/_private/workers/default_worker.py", line 289 in <module>
(RayWorkerWrapper pid=24036)
(RayWorkerWrapper pid=24036) Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, zstandard.backend_c, charset_normalizer.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, sentencepiece._sentencepiece, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, pyarrow.lib, pyarrow._json, PIL._imaging, zmq.backend.cython._zmq (total: 42)
(RayWorkerWrapper pid=24649)
(RayWorkerWrapper pid=24649)
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffed01258d58da257058d7c65e02000000 Worker ID: dfeb342e54f14dc8d324f5c0ca0e7c5173847f6ae45daf12ba3ad0e9 Node ID: 3474bf626ca27e7ff63f0281c7ac592f54f452c528caf1c9294256ce Worker IP address: 192.168.201.210 Worker port: 10403 Worker PID: 24649 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(RayWorkerWrapper pid=2844, ip=192.168.201.208) INFO 07-25 10:45:55 model_runner.py:980] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. [repeated 14x across cluster]
(RayWorkerWrapper pid=2844, ip=192.168.201.208) INFO 07-25 10:45:55 model_runner.py:984] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. [repeated 14x across cluster]
I am actively monitoring CPU memory usage, and it is not even close to the capacity. Both servers have 1.5TB, and this script consumes ~50GBs and then crashes. I have also tried reducing batch-size to 1, but didn’t help.
I would appreciate any feedback on what is happening under the hood here, especially since the model is 8B with low batch-size and low seq-length, which runs just fine even on a single H100 GPU.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 22 (16 by maintainers)
Usually
--enforce-eagerwill slow down the inference. But it seems this might be some NCCL bug, when using allreduce across nodes with cudagraph, it will cause problem. So we have to use--enforce-eager.