uccl: Binding error when running nccl-tests with multi-node UCCL in Docker
When I run the 16-GPU NCCL tests across two machines that use UCCL in the containers, an error is reported: Check failed: bind(listen_fd, (struct sockaddr)&serv_addr, sizeof(serv_addr)) >= 0 ERROR: binding.
error info:
F20250720 17:52:37.391902 1125343 util.h:139] Check failed: bind(*listen_fd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) >= 0 ERROR: binding
*** Check failure stack trace: ***
[n232-194-226:1124750] *** Process received signal ***
[n232-194-226:1124750] Signal: Aborted (6)
[n232-194-226:1124750] Signal code: (-6)
[n232-194-226:1124750] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7ff972cbe330]
[n232-194-226:1124750] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7ff972d17b2c]
[n232-194-226:1124750] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7ff972cbe27e]
[n232-194-226:1124750] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7ff972ca18ff]
[n232-194-226:1124750] [ 4] /lib/x86_64-linux-gnu/libglog.so.1(+0x10031)[0x7ff6e8a72031]
[n232-194-226:1124750] [ 5] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage9SendToLogEv+0xb33)[0x7ff6e8a79463]
[n232-194-226:1124750] [ 6] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage5FlushEv+0xd8)[0x7ff6e8a71b78]
[n232-194-226:1124750] [ 7] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google15LogMessageFatalD2Ev+0xd)[0x7ff6e8a7e66d]
[n232-194-226:1124750] [ 8] /workspace/uccl/rdma/libnccl-net-uccl.so(+0x3eff4)[0x7ff6e8ae6ff4]
[n232-194-226:1124750] [ 9] /lib/x86_64-linux-gnu/libc.so.6(+0xa1ed3)[0x7ff972d1aed3]
[n232-194-226:1124750] [10] /workspace/uccl/rdma/libnccl-net-uccl.so(_ZN4uccl12RDMAEndpoint24initialize_engine_by_devEi+0xa7)[0x7ff6e8ad0047]
[n232-194-226:1124750] [11] /workspace/uccl/rdma/libnccl-net-uccl.so(_Z12pluginListeniPvPS_+0x5a)[0x7ff6e8abef8a]
[n232-194-226:1124750] [12] /lib/x86_64-linux-gnu/libnccl.so.2(+0xbdb12)[0x7ff9731c6b12]
[n232-194-226:1124750] [13] /lib/x86_64-linux-gnu/libnccl.so.2(+0x76fe7)[0x7ff97317ffe7]
[n232-194-226:1124750] [14] /lib/x86_64-linux-gnu/libnccl.so.2(+0x7874f)[0x7ff97318174f]
[n232-194-226:1124750] [15] /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x7ff972d15aa4]
[n232-194-226:1124750] [16] /lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x7ff972da2a34]
[n232-194-226:1124750] *** End of error message ***
F20250720 17:52:37.395483 598393 util.h:139] Check failed: bind(*listen_fd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)) >= 0 ERROR: binding
*** Check failure stack trace: ***
[n232-194-227:597785] *** Process received signal ***
[n232-194-227:597785] Signal: Aborted (6)
[n232-194-227:597785] Signal code: (-6)
[n232-194-227:597785] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7fbdc30be330]
[n232-194-227:597785] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c)[0x7fbdc3117b2c]
[n232-194-227:597785] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e)[0x7fbdc30be27e]
[n232-194-227:597785] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xdf)[0x7fbdc30a18ff]
[n232-194-227:597785] [ 4] /lib/x86_64-linux-gnu/libglog.so.1(+0x10031)[0x7fbb382c2031]
[n232-194-227:597785] [ 5] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage9SendToLogEv+0xb33)[0x7fbb382c9463]
[n232-194-227:597785] [ 6] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google10LogMessage5FlushEv+0xd8)[0x7fbb382c1b78]
[n232-194-227:597785] [ 7] /lib/x86_64-linux-gnu/libglog.so.1(_ZN6google15LogMessageFatalD2Ev+0xd)[0x7fbb382ce66d]
[n232-194-227:597785] [ 8] /workspace/uccl/rdma/libnccl-net-uccl.so(+0x3eff4)[0x7fbb38336ff4]
[n232-194-227:597785] [ 9] /lib/x86_64-linux-gnu/libc.so.6(+0xa1ed3)[0x7fbdc311aed3]
[n232-194-227:597785] [10] /workspace/uccl/rdma/libnccl-net-uccl.so(_ZN4uccl12RDMAEndpoint24initialize_engine_by_devEi+0xa7)[0x7fbb38320047]
[n232-194-227:597785] [11] /workspace/uccl/rdma/libnccl-net-uccl.so(_Z12pluginListeniPvPS_+0x5a)[0x7fbb3830ef8a]
[n232-194-227:597785] [12] /lib/x86_64-linux-gnu/libnccl.so.2(+0xbdb12)[0x7fbdc35c6b12]
[n232-194-227:597785] [13] /lib/x86_64-linux-gnu/libnccl.so.2(+0x76fe7)[0x7fbdc357ffe7]
[n232-194-227:597785] [14] /lib/x86_64-linux-gnu/libnccl.so.2(+0x7874f)[0x7fbdc358174f]
[n232-194-227:597785] [15] /lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x7fbdc3115aa4]
[n232-194-227:597785] [16] /lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x7fbdc31a2a34]
[n232-194-227:597785] *** End of error message ***
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 597785 on node node-0 exited on signal 6 (Aborted).
--------------------------------------------------------------------------
command :
UCCL_HOME=/workspace/uccl OMPI_ALLOW_RUN_AS_ROOT=1 OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 ./run_nccl_test.sh uccl 16 1 0 8
run_nccl_test.sh:
# !/bin/bash
source ../scripts/shared.sh
# Usage ./run_nccl_test.sh [UCCL] [# of Nodes] [# of GPUs per process] [allreduce/alltoall: 0/1]
TEST=${1:-1}
NUM_PROCS=${2:-2}
NUM_GPUS_PER_PROC=${3:-8}
PROG_OPTION=${4:-0}
PROCS_PER_NODE=${5:-1}
HOSTNAME=${6:-"hosts"}
HOSTFILE="$HOSTNAME"
NODES=""
while IFS= read -r line || [[ -n "$line" ]]; do
[[ -z "$line" || "$line" =~ ^# ]] && continue
host=$(echo "$line" | awk '{print $1}')
NODES="${NODES}${host},"
done < "$HOSTFILE"
# Trim trailing comma
NODES=${NODES%,}
echo "Parsed NODES: $NODES"
IFS=',' read -ra ADDR <<< "$NODES"
for node in "${ADDR[@]}"; do
echo "Checking GPU usage on $node..."
gpu_procs=$(ssh "$node" "nvidia-smi --query-compute-apps=pid --format=csv,noheader")
if [[ -n "$gpu_procs" ]]; then
echo "GPU in use on $node by PIDs: $gpu_procs"
exit 1
else
echo "No GPU processes found on $node."
fi
done
# all_gather_perf all_reduce_perf alltoall_perf broadcast_perf gather_perf
# hypercube_perf reduce_perf reduce_scatter_perf scatter_perf sendrecv_perf
if [ "$PROG_OPTION" -eq 0 ]; then
PROG_NAME=all_reduce_perf
# We force allreduce to use Simple protocol to avoid outlier for both UCCL and NCCL.
NCCL_PROTO="Simple"
elif [ "$PROG_OPTION" -eq 1 ]; then
PROG_NAME=alltoall_perf
else
PROG_NAME=sendrecv_perf
fi
if [ "$TEST" = "nccl" ]; then
echo "Running NCCL test"
plugin_path=""
elif [ "$TEST" = "uccl" ]; then
echo "Running UCCL test"
# plugin_path=`python -c "import uccl; print(uccl.nccl_plugin_path())"`
plugin_path="${UCCL_HOME}/rdma/libnccl-net-uccl.so"
echo "plugin_path: ${plugin_path}"
else
echo "Unsupport benchmark type."
exit 1
fi
HCA_NAMES="mlx5_0:1,mlx5_2:1,mlx5_4:1,mlx5_5:1"
mpirun --allow-run-as-root -np ${NUM_PROCS} -N ${PROCS_PER_NODE} \
-x NCCL_IB_DISABLE=0 \
-x NCCL_IB_HCA=${HCA_NAMES}\
-x NCCL_IB_GID_INDEX=3 \
-x CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
-x NCCL_DEBUG=WARN \
-x NCCL_NET_PLUGIN=$plugin_path \
--mca btl tcp,self \
--mca btl_tcp_if_include eth0 \
-x UCCL_IB_HCA=${HCA_NAMES} \
--mca coll_hcoll_enable 0 \
-x NCCL_NET_GDR_LEVEL=0 \
-x NCCL_SOCKET_IFNAME=eth0 \
-x UCCL_SOCKET_IFNAME=eth0 \
-x NCCL_IB_GID_INDEX=3 \
-hostfile ${HOSTNAME} \
${UCCL_HOME}/thirdparty/nccl-tests/build/${PROG_NAME} -c 0 \
-b 1K -e 1G \
-f 2 -w 50 -n 50 \
-g 1 -t ${NUM_GPUS_PER_PROC}
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 33 (16 by maintainers)
Hi @hjx620 Thanks for all these tests! I checked the log, and found that all errors are related to the sharing NIC issue.
Check failed: rdma_ctx_map_.find(ctrl_work.peer_id) == rdma_ctx_map_.end()This check will fail when multiple processes (GPUs) share the same NIC, it is expected!If you are interested, I think you can take a look at why the NIC-GPU mapping automatically detected by UCCL is incorrect. Your contribution is welcome!
after deleting
-x UCCL_IB_GID_IDX=3 \,the hang changes to errorCheck failed: rdma_ctx_map_.find(ctrl_work.peer_id) == rdma_ctx_map_.end()and log
uccl-log
From the log, i see that two different process(thus different GPUs) listen on the same dev. e.g.,
topo
@hjx620 Can you pull the last commit from the main branch, rebuild the UCCL plugin, and run the test again?
It seems that we accidentally enabled listening on some hardcoded socket ports for testing purposes. The latest commit should fix it, and I have tested on my local servers with similar hca list
rdma0:1,rdma2:1,rdma4:1,rdma5:1.