verifiers: Cryptic NCCL errors

This is my server command

MPLBACKEND="agg"  CUDA_VISIBLE_DEVICES=0,1  NCCL_DEBUG=INFO  vf-vllm --model 'Qwen/Qwen2.5-1.5B-Instruct' --tensor-parallel-size 2 --gpu-memory-utilization 0.5

My GRPOTrainer connects and this is the result:

INFO 06-07 17:34:08 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 06-07 17:34:09 [__init__.py:239] Automatically detected platform cuda.
─────────────────────── 🧠 Symbolic Compressor Training ────────────────────────
Base model: Qwen/Qwen2.5-1.5B-Instruct
Stages: identity → structured → freeform → cognition
────────────────────────────────────────────────────────────────────────────────

───────────────────────────── Stage 1/4: IDENTITY ──────────────────────────────
Identity Pretraining: Learn basic compress → decompress → match
α=0.005 β=3.0
────────────────────────────────────────────────────────────────────────────────

📊 Loading dataset...
✓ Wikipedia dataset loaded
✓ Split: 1450 train, 50 eval

🏗️ Setting up environment...
✓ Environment ready

🚀 Training...
📦 Loading model: Qwen/Qwen2.5-1.5B-Instruct
Using Liger kernel
Applied Liger kernels to Qwen2
✓ Model loaded

⚙️ Configuring training...
                                                      
  Alpha (compression)         0.005                   
  Beta (fidelity)             3.0                     
  Max steps                   150                     
  Training samples            1450                    
  Batch size                  2 prompts → 4 rollouts  
                                                      

🏋️ Creating trainer...
[2025-06-07 17:34:15,257] [INFO] [real_accelerator.py:254:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2025-06-07 17:34:16 - verifiers.trainers.grpo_trainer - INFO - Filtering dataset for prompts with length <= 1024
2025-06-07 17:34:17 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:19 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:21 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:23 - verifiers.inference.vllm_client - INFO - Server is not up yet. Retrying in 2.0 seconds...
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - Server is up!
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Starting init_communicator
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Getting world size from http://0.0.0.0:8000/get_world_size/
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] World size response: status=200
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] vLLM world size: 2
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Client rank: 2, total world size: 3
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Sending init_communicator request to http://0.0.0.0:8000/init_communicator/
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Init communicator response: status=200
2025-06-07 17:34:25 - verifiers.inference.vllm_client - INFO - [VLLM_CLIENT] Initializing PyNcclCommunicator on device 0, rank 2, world_size 3
INFO 06-07 17:34:25 [utils.py:1055] Found nccl from library libnccl.so.2
INFO 06-07 17:34:25 [pynccl.py:69] vLLM is using nccl==2.21.5

25ef2a5020cf:14816:14816 [0] init.cc:943 NCCL WARN Duplicate GPU detected : rank 2 and rank 0 both on CUDA device 25000
─────────────────────────────── ❌ Stage Failed ────────────────────────────────
❌ Stage identity failed: NCCL error: invalid usage (run with NCCL_DEBUG=WARN 
for details)
🛑 Stopping training
────────────────────────────────────────────────────────────────────────────────
[17:34:26] ERROR    Training failed: NCCL error: invalid usage (run with        
                    NCCL_DEBUG=WARN for details)                                
Traceback (most recent call last):
  File "/thauten/train_compressor.py", line 846, in <module>
    main()
  File "/thauten/train_compressor.py", line 820, in main
    current_model_path = train_stage(stage, compressor_env, current_model_path, base_model_name)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/thauten/train_compressor.py", line 711, in train_stage
    trainer = vf.GRPOTrainer(
              ^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/verifiers/trainers/grpo_trainer.py", line 297, in __init__
    self.vllm_client.init_communicator()
  File "/venv/main/lib/python3.12/site-packages/verifiers/inference/vllm_client.py", line 177, in init_communicator
    self.pynccl_comm = PyNcclCommunicator(pg, device=device)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
    self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
    self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
  File "/venv/main/lib/python3.12/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
    raise RuntimeError(f"NCCL error: {error_str}")
RuntimeError: NCCL error: invalid usage (run with NCCL_DEBUG=WARN for details)

Is verifiers incompatible with nccl 2.21.5 ?

The training code is over here if necessary https://github.com/holo-q/thauten/blob/master/train_compressor.py

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 3
  • Comments: 18 (7 by maintainers)

Most upvoted comments

Just wanted to share something that worked for me

setting export NCCL_P2P_DISABLE=1 and export NCCL_IB_DISABLE=1 in both terminals

I had assumed setting it in one terminal would apply globally across the virtual environment 🥲, but it only worked after I explicitly set it in both