uccl: Binding error when running nccl-tests with multi-node UCCL in Docker

When I run the 16-GPU NCCL tests across two machines that use UCCL in the containers, an error is reported: Check failed: bind(listen_fd, (struct sockaddr)&serv_addr, sizeof(serv_addr)) >= 0 ERROR: binding.

error info:

F20250720 17:52:37.391902 1125343 util.h:139] Check failed: bind(*listen_fd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) >= 0 ERROR: binding
*** Check failure stack trace: ***
[n232-194-226:1124750] *** Process received signal ***
[n232-194-226:1124750] Signal: Aborted (6)
[n232-194-226:1124750] Signal code:  (-6)
[n232-194-226:1124750] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7ff972cbe330]
[n232-194-226:1124750] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7ff972d17b2c]
[n232-194-226:1124750] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7ff972cbe27e]
[n232-194-226:1124750] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7ff972ca18ff]
[n232-194-226:1124750] [ 4] /lib/x86_64-linux-gnu/libglog.so.1(+0x10031)[0x7ff6e8a72031]
[n232-194-226:1124750] [ 5] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage9SendToLogEv+0xb33)[0x7ff6e8a79463]
[n232-194-226:1124750] [ 6] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage5FlushEv+0xd8)[0x7ff6e8a71b78]
[n232-194-226:1124750] [ 7] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google15LogMessageFatalD2Ev+0xd)[0x7ff6e8a7e66d]
[n232-194-226:1124750] [ 8] /workspace/uccl/rdma/libnccl-net-uccl.so(+0x3eff4)[0x7ff6e8ae6ff4]
[n232-194-226:1124750] [ 9] /lib/x86_64-linux-gnu/libc.so.6(+0xa1ed3)[0x7ff972d1aed3]
[n232-194-226:1124750] [10] /workspace/uccl/rdma/libnccl-net-uccl.so(_ZN4uccl12RDMAEndpoint24initialize_engine_by_devEi+0xa7)[0x7ff6e8ad0047]
[n232-194-226:1124750] [11] /workspace/uccl/rdma/libnccl-net-uccl.so(_Z12pluginListeniPvPS_+0x5a)[0x7ff6e8abef8a]
[n232-194-226:1124750] [12] /lib/x86_64-linux-gnu/libnccl.so.2(+0xbdb12)[0x7ff9731c6b12]
[n232-194-226:1124750] [13] /lib/x86_64-linux-gnu/libnccl.so.2(+0x76fe7)[0x7ff97317ffe7]
[n232-194-226:1124750] [14] /lib/x86_64-linux-gnu/libnccl.so.2(+0x7874f)[0x7ff97318174f]
[n232-194-226:1124750] [15] /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x7ff972d15aa4]
[n232-194-226:1124750] [16] /lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x7ff972da2a34]
[n232-194-226:1124750] *** End of error message ***
F20250720 17:52:37.395483 598393 util.h:139] Check failed: bind(*listen_fd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) >= 0 ERROR: binding
*** Check failure stack trace: ***
[n232-194-227:597785] *** Process received signal ***
[n232-194-227:597785] Signal: Aborted (6)
[n232-194-227:597785] Signal code:  (-6)
[n232-194-227:597785] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7fbdc30be330]
[n232-194-227:597785] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7fbdc3117b2c]
[n232-194-227:597785] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7fbdc30be27e]
[n232-194-227:597785] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7fbdc30a18ff]
[n232-194-227:597785] [ 4] /lib/x86_64-linux-gnu/libglog.so.1(+0x10031)[0x7fbb382c2031]
[n232-194-227:597785] [ 5] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage9SendToLogEv+0xb33)[0x7fbb382c9463]
[n232-194-227:597785] [ 6] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage5FlushEv+0xd8)[0x7fbb382c1b78]
[n232-194-227:597785] [ 7] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google15LogMessageFatalD2Ev+0xd)[0x7fbb382ce66d]
[n232-194-227:597785] [ 8] /workspace/uccl/rdma/libnccl-net-uccl.so(+0x3eff4)[0x7fbb38336ff4]
[n232-194-227:597785] [ 9] /lib/x86_64-linux-gnu/libc.so.6(+0xa1ed3)[0x7fbdc311aed3]
[n232-194-227:597785] [10] /workspace/uccl/rdma/libnccl-net-uccl.so(_ZN4uccl12RDMAEndpoint24initialize_engine_by_devEi+0xa7)[0x7fbb38320047]
[n232-194-227:597785] [11] /workspace/uccl/rdma/libnccl-net-uccl.so(_Z12pluginListeniPvPS_+0x5a)[0x7fbb3830ef8a]
[n232-194-227:597785] [12] /lib/x86_64-linux-gnu/libnccl.so.2(+0xbdb12)[0x7fbdc35c6b12]
[n232-194-227:597785] [13] /lib/x86_64-linux-gnu/libnccl.so.2(+0x76fe7)[0x7fbdc357ffe7]
[n232-194-227:597785] [14] /lib/x86_64-linux-gnu/libnccl.so.2(+0x7874f)[0x7fbdc358174f]
[n232-194-227:597785] [15] /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x7fbdc3115aa4]
[n232-194-227:597785] [16] /lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x7fbdc31a2a34]
[n232-194-227:597785] *** End of error message ***
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 597785 on node node-0 exited on signal 6 (Aborted).
--------------------------------------------------------------------------

command :

UCCL_HOME=/workspace/uccl OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 ./run_nccl_test.sh uccl 16 1 0 8

run_nccl_test.sh:

# !/bin/bash

source ../scripts/shared.sh

# Usage ./run_nccl_test.sh [UCCL] [# of Nodes] [# of GPUs per process] [allreduce/alltoall: 0/1]

TEST=${1:-1}
NUM_PROCS=${2:-2}
NUM_GPUS_PER_PROC=${3:-8}
PROG_OPTION=${4:-0}
PROCS_PER_NODE=${5:-1}
HOSTNAME=${6:-"hosts"}

HOSTFILE="$HOSTNAME"
NODES=""
while IFS= read -r line || [[ -n "$line" ]]; do
  [[ -z "$line" || "$line" =~ ^# ]] && continue
  host=$(echo "$line" | awk '{print $1}')
  NODES="${NODES}${host},"
done < "$HOSTFILE"

# Trim trailing comma
NODES=${NODES%,}
echo "Parsed NODES: $NODES"

IFS=',' read -ra ADDR <<< "$NODES"
for node in "${ADDR[@]}"; do
    echo "Checking GPU usage on $node..."
    gpu_procs=$(ssh "$node" "nvidia-smi --query-compute-apps=pid --format=csv,noheader")

    if [[ -n "$gpu_procs" ]]; then
        echo "GPU in use on $node by PIDs: $gpu_procs"
        exit 1
    else
        echo "No GPU processes found on $node."
    fi
done

# all_gather_perf  all_reduce_perf  alltoall_perf  broadcast_perf  gather_perf
# hypercube_perf  reduce_perf  reduce_scatter_perf  scatter_perf  sendrecv_perf

if [ "$PROG_OPTION" -eq 0 ]; then
    PROG_NAME=all_reduce_perf
    # We force allreduce to use Simple protocol to avoid outlier for both UCCL and NCCL.
    NCCL_PROTO="Simple"
elif [ "$PROG_OPTION" -eq 1 ]; then
    PROG_NAME=alltoall_perf
else
    PROG_NAME=sendrecv_perf
fi

if [ "$TEST" = "nccl" ]; then
    echo "Running NCCL test"
    plugin_path=""
elif [ "$TEST" = "uccl" ]; then
    echo "Running UCCL test"
    # plugin_path=`python -c "import uccl; print(uccl.nccl_plugin_path())"`
    plugin_path="${UCCL_HOME}/rdma/libnccl-net-uccl.so"
    echo "plugin_path: ${plugin_path}"
else
    echo "Unsupport benchmark type."
    exit 1
fi

HCA_NAMES="mlx5_0:1,mlx5_2:1,mlx5_4:1,mlx5_5:1"

mpirun --allow-run-as-root -np ${NUM_PROCS} -N ${PROCS_PER_NODE} \
    -x NCCL_IB_DISABLE=0  \
    -x NCCL_IB_HCA=${HCA_NAMES}\
    -x NCCL_IB_GID_INDEX=3 \
    -x CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
    -x NCCL_DEBUG=WARN \
    -x NCCL_NET_PLUGIN=$plugin_path \
    --mca btl tcp,self \
    --mca btl_tcp_if_include eth0 \
    -x UCCL_IB_HCA=${HCA_NAMES} \
    --mca coll_hcoll_enable 0 \
    -x NCCL_NET_GDR_LEVEL=0 \
    -x NCCL_SOCKET_IFNAME=eth0 \
    -x UCCL_SOCKET_IFNAME=eth0 \
    -x NCCL_IB_GID_INDEX=3 \
    -hostfile ${HOSTNAME} \
    ${UCCL_HOME}/thirdparty/nccl-tests/build/${PROG_NAME} -c 0 \
    -b 1K -e 1G \
    -f 2 -w 50 -n 50 \
    -g 1 -t ${NUM_GPUS_PER_PROC}

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 33 (16 by maintainers)

Most upvoted comments

Hi @hjx620 Thanks for all these tests! I checked the log, and found that all errors are related to the sharing NIC issue. Check failed: rdma_ctx_map_.find(ctrl_work.peer_id) == rdma_ctx_map_.end() This check will fail when multiple processes (GPUs) share the same NIC, it is expected!

If you are interested, I think you can take a look at why the NIC-GPU mapping automatically detected by UCCL is incorrect. Your contribution is welcome!

after deleting -x UCCL_IB_GID_IDX=3 \ ,the hang changes to error Check failed: rdma_ctx_map_.find(ctrl_work.peer_id) == rdma_ctx_map_.end()

and log

uccl-log

From the log, i see that two different process(thus different GPUs) listen on the same dev. e.g.,

I20250721 07:39:16.015934  4148 nccl_plugin.cc:327] [Plugin] Listen on dev: 3 from PID: 3842
I20250721 07:39:16.015937  4145 nccl_plugin.cc:327] [Plugin] Listen on dev: 3 from PID: 3843

topo

root@n232-194-227:/workspace/uccl/rdma# nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	NIC8	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV8	NV8	NV8	NV8	NV8	NV8	NV8	NODE	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU1	NV8	 X 	NV8	NV8	NV8	NV8	NV8	NV8	NODE	NODE	PIX	NODE	NODE	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU2	NV8	NV8	 X 	NV8	NV8	NV8	NV8	NV8	NODE	NODE	NODE	PIX	NODE	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU3	NV8	NV8	NV8	 X 	NV8	NV8	NV8	NV8	NODE	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	0-47,96-143	0	N/A
GPU4	NV8	NV8	NV8	NV8	 X 	NV8	NV8	NV8	SYS	SYS	SYS	SYS	SYS	PIX	NODE	NODE	NODE	48-95,144-191	1	N/A
GPU5	NV8	NV8	NV8	NV8	NV8	 X 	NV8	NV8	SYS	SYS	SYS	SYS	SYS	NODE	PIX	NODE	NODE	48-95,144-191	1	N/A
GPU6	NV8	NV8	NV8	NV8	NV8	NV8	 X 	NV8	SYS	SYS	SYS	SYS	SYS	NODE	NODE	PIX	NODE	48-95,144-191	1	N/A
GPU7	NV8	NV8	NV8	NV8	NV8	NV8	NV8	 X 	SYS	SYS	SYS	SYS	SYS	NODE	NODE	NODE	PIX	48-95,144-191	1	N/A
NIC0	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	 X 	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS				
NIC1	NODE	NODE	NODE	NODE	SYS	SYS	SYS	SYS	PIX	 X 	NODE	NODE	NODE	SYS	SYS	SYS	SYS				
NIC2	NODE	PIX	NODE	NODE	SYS	SYS	SYS	SYS	NODE	NODE	 X 	NODE	NODE	SYS	SYS	SYS	SYS				
NIC3	NODE	NODE	PIX	NODE	SYS	SYS	SYS	SYS	NODE	NODE	NODE	 X 	NODE	SYS	SYS	SYS	SYS				
NIC4	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	NODE	NODE	NODE	NODE	 X 	SYS	SYS	SYS	SYS				
NIC5	SYS	SYS	SYS	SYS	PIX	NODE	NODE	NODE	SYS	SYS	SYS	SYS	SYS	 X 	NODE	NODE	NODE				
NIC6	SYS	SYS	SYS	SYS	NODE	PIX	NODE	NODE	SYS	SYS	SYS	SYS	SYS	NODE	 X 	NODE	NODE				
NIC7	SYS	SYS	SYS	SYS	NODE	NODE	PIX	NODE	SYS	SYS	SYS	SYS	SYS	NODE	NODE	 X 	NODE				
NIC8	SYS	SYS	SYS	SYS	NODE	NODE	NODE	PIX	SYS	SYS	SYS	SYS	SYS	NODE	NODE	NODE	 X 				

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8


@hjx620 Can you pull the last commit from the main branch, rebuild the UCCL plugin, and run the test again?

It seems that we accidentally enabled listening on some hardcoded socket ports for testing purposes. The latest commit should fix it, and I have tested on my local servers with similar hca list rdma0:1,rdma2:1,rdma4:1,rdma5:1.