verifiers: Cryptic NCCL errors
This is my server command
MPLBACKEND="agg" CUDA_VISIBLE_DEVICES=0,1 NCCL_DEBUG=INFO vf-vllm --model 'Qwen/Qwen2.5-1.5B-Instruct' --tensor-parallel-size 2 --gpu-memory-utilization 0.5
My GRPOTrainer connects and this is the result:
INFO 06-07 17:34:08 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 06-07 17:34:09 [__init__.py:239] Automatically detected platform cuda.
─────────────────────── 🧠 Symbolic Compressor Training ────────────────────────
Base model: Qwen/Qwen2.5-1.5B-Instruct
Stages: identity → structured → freeform → cognition
────────────────────────────────────────────────────────────────────────────────
───────────────────────────── Stage 1/4: IDENTITY ──────────────────────────────
Identity Pretraining: Learn basic compress → decompress → match
α=0.005 β=3.0
────────────────────────────────────────────────────────────────────────────────
📊 Loading dataset...
✓ Wikipedia dataset loaded
✓ Split: 1450 train, 50 eval
🏗️ Setting up environment...
✓ Environment ready
🚀 Training...
📦 Loading model: Qwen/Qwen2.5-1.5B-Instruct
Using Liger kernel
Applied Liger kernels to Qwen2
✓ Model loaded
⚙️ Configuring training...
Alpha (compression) 0.005
Beta (fidelity) 3.0
Max steps 150
Training samples 1450
Batch size 2 prompts → 4 rollouts
🏋️ Creating trainer...
[2025-06-07 17:34:15,257] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2025-06-07 17:34:16 - verifiers.trainers.grpo_trainer - INFO - Filtering dataset for prompts with length <= 1024
2025-06-07 17:34:17 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:19 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:21 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:23 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - Server is up!
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Starting init_communicator
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Getting world size from http://0.0.0.0:8000/get_world_size/
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] World size response: status=200
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] vLLM world size: 2
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Client rank: 2, total world size: 3
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Sending init_communicator request to http://0.0.0.0:8000/init_communicator/
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Init communicator response: status=200
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Initializing PyNcclCommunicator on device 0, rank 2, world_size 3
INFO 06-07 17:34:25 [utils.py:1055] Found nccl from library libnccl.so.2
INFO 06-07 17:34:25 [pynccl.py:69] vLLM is using nccl==2.21.5
25ef2a5020cf:14816:14816 [0] init.cc:943 NCCL WARN Duplicate GPU detected : rank 2 and rank 0 both on CUDA device 25000
─────────────────────────────── ❌ Stage Failed ────────────────────────────────
❌ Stage identity failed: NCCL error: invalid usage (run with NCCL_DEBUG=WARN
for details)
🛑 Stopping training
────────────────────────────────────────────────────────────────────────────────
[17:34:26] ERROR Training failed: NCCL error: invalid usage (run with
NCCL_DEBUG=WARN for details)
Traceback (most recent call last):
File "/thauten/train_compressor.py", line 846, in <module>
main()
File "/thauten/train_compressor.py", line 820, in main
current_model_path = train_stage(stage, compressor_env, current_model_path, base_model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/thauten/train_compressor.py", line 711, in train_stage
trainer = vf.GRPOTrainer(
^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/verifiers/trainers/grpo_trainer.py", line 297, in __init__
self.vllm_client.init_communicator()
File "/venv/main/lib/python3.12/site-packages/verifiers/inference/vllm_client.py", line 177, in init_communicator
self.pynccl_comm = PyNcclCommunicator(pg, device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/venv/main/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
File "/venv/main/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
raise RuntimeError(f"NCCL error: {error_str}")
RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details)
Is verifiers incompatible with nccl 2.21.5 ?
The training code is over here if necessary https://github.com/holo-q/thauten/blob/master/train_compressor.py
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 3
- Comments: 18 (7 by maintainers)
Just wanted to share something that worked for me
setting export NCCL_P2P_DISABLE=1 and export NCCL_IB_DISABLE=1 in both terminals
I had assumed setting it in one terminal would apply globally across the virtual environment 🥲, but it only worked after I explicitly set it in both