transformers: Excessive GPU-GPU communication with GPT2 making multi-GPU training slow?

ok, a quick hack to add ratios relative to 1gpu, so now it’s easier to see the comparison.

perl -lne 'BEGIN{ print qq[|model|block|type|runtime|sample/sec|ratios]; print "|-" x 6, "|"} $d=qr/([\d\.]+)/; if (m|^(\S+) $d (\S+) ..train_runtime.. $d, .train_samples_per_second.. $d|) {if($3=="1GPU") {$s=$4; print "| " x 6, "|"}; print qq[|$1|$2|$3|] . int($4). "|". sprintf("%0.1f", $5)."|".sprintf("%0.1f", $4/$s)."|"}'  log.txt

So I added a new column runtime ratios and each 4 rows get recalculated wrt to their first runtime entry with 1gpu.

edit: someone asked to explain the ratio and why the runtime is faster for DDP, but samples per second is smaller.

Here is a puzzle to solve:

1. one cake eater eats the cake at 60 sec/cake
2. now a second cake eater joins and who eats at the same speed as the first one, but now after every bite they have to shout “ML rocks”, which slows down both of them, so they are now eating 20% slower than when alone

Will one cake eater finish the cake faster than two of them?

(the answer is after the table, so you don’t see it right away)

and the answer to the puzzle posted at the beginning of this comment: 2 cake eaters will eat the cake faster together despite the slowdown, because they only have half a cake to finish each!

Same here, while each of the GPUs in the DDP assembly performs slower due to the gradient syncing, but because it has to consume half the samples, overall the assembly will train faster.

Further, this benchmark is just for 2 GPUs

So going from 1GPU to 2GPUs, you create the overhead, and so you get some loss in performance, and some gain

When you go from 2GPUs to 4GPUs (on the same node), it’s pure performance doubling. i.e. 4GPUs will perform disproportionally faster than 2GPUs over 1 GPU.

• 1 GPU has no inter-gpu communication to do
• 2+ gpus have to average gradients

so they add this overhead, but then they can parallelize the processing so the overhead becomes almost negligible as the number of GPUs grows

The next problem is once you outgrow a single node. So the next issue is inter-node connects, which can be blazing fast (Infiniband) or super-slow (ethernet hub). So to scale from 8GPUs to 10 (for 8-gpu node), you first lose performance, because now the inter-node connection is the slow component that slows everything down. But as you add more nodes, again that overhead becomes less and less significant.

Of course when working with multi-node one often uses other parallelization techniques than DDP, so it’s PP or TP (https://huggingface.co/transformers/parallelism.html#concepts), and there one generally performs TP only inside a node, and PP and DP over nodes.

It’d be amazing if someone re-did this table for 1, 2, 4 gpus, then 1, 2, 4 nodes.

OK, now we have some extensive benchmarks for the RTX8000 machine. This machine does not have NVLink, but it apparently can do P2P GPU-GPU communication via the PCI bus. However, this seems to be quite slow – slower, in fact, than disabling P2P altogether.

Here’s nvidia-smi topo -m:

        GPU0    GPU1    GPU2    GPU3    mlx5_0  CPU Affinity    NUMA Affinity
GPU0     X      SYS     SYS     SYS     SYS     0-7     0-1
GPU1    SYS      X      SYS     SYS     SYS     0-7     0-1
GPU2    SYS     SYS      X      SYS     SYS     0-7     0-1
GPU3    SYS     SYS     SYS      X      SYS     0-7     0-1
mlx5_0  SYS     SYS     SYS     SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

I used the script from before (slightly expanded) and set max-steps to 800 for the single GPU case, 400 for two GPUs, and 200 for 4 GPUs. Here are the benchmarks (long!):

I managed to get some time on a node with 4x V100s. For the Large model, it gets 3.83s/it with an ETA of 1248:01:43 (!).

Here’s the output of p2pBandwidthLatencyTest on the V100 system:

[bd52@gv02 p2pBandwidthLatencyTest]$ ./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, Tesla V100-PCIE-32GB, pciBusID: 6, pciDeviceID: 0, pciDomainID:0
Device: 1, Tesla V100-PCIE-32GB, pciBusID: 2f, pciDeviceID: 0, pciDomainID:0
Device: 2, Tesla V100-PCIE-32GB, pciBusID: 86, pciDeviceID: 0, pciDomainID:0
Device: 3, Tesla V100-PCIE-32GB, pciBusID: d8, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0       1     1     1     1
     1       1     1     1     1
     2       1     1     1     1
     3       1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 768.57  11.42  11.52  11.53 
     1  11.39 770.46  11.50  11.53 
     2  11.42  11.43 771.22  11.45 
     3  11.42  11.43  11.44 769.70 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3 
     0 767.06   9.93   9.68   9.49 
     1   9.93 769.33   9.33   9.50 
     2   9.87   9.35 769.70  10.05 
     3   9.66   9.68   9.92 770.08 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 771.22  15.98  16.04  16.16 
     1  16.00 773.51  16.11  16.07 
     2  15.90  15.99 772.75  15.83 
     3  16.05  16.01  15.85 772.55 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 770.84  18.72  18.41  18.07 
     1  18.52 772.94  18.82  18.30 
     2  18.41  18.16 771.80  19.13 
     3  18.40  17.99  18.94 771.22 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3 
     0   1.89  14.77  14.42  14.59 
     1  14.52   1.91  15.50  15.50 
     2  15.53  15.42   1.87  14.44 
     3  14.76  14.71  14.51   1.82 

   CPU     0      1      2      3 
     0   2.52   8.33   8.61   8.55 
     1   8.20   2.49   8.50   8.49 
     2   8.30   8.29   2.61   8.69 
     3   8.41   8.36   8.74   2.56 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3 
     0   1.86   1.60   1.65   1.64 
     1   1.59   1.91   1.64   1.65 
     2   1.65   1.63   1.88   1.58 
     3   1.65   1.64   1.59   1.82 

   CPU     0      1      2      3 
     0   2.51   2.05   2.02   2.02 
     1   2.14   2.54   2.04   2.02 
     2   2.28   2.18   2.61   2.18 
     3   2.32   2.19   2.24   2.73 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

And for comparison, here’s the dual 3090 w/NVLINK system:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, GeForce RTX 3090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, GeForce RTX 3090, pciBusID: 21, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0       1     1
     1       1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 831.56  11.25 
     1  11.33 831.12 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1 
     0 810.85  52.77 
     1  52.85 832.89 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 812.31  16.55 
     1  16.75 838.03 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 821.29 101.41 
     1 101.80 835.34 
P2P=Disabled Latency Matrix (us)
   GPU     0      1 
     0   1.59  33.13 
     1  20.55   1.48 

   CPU     0      1 
     0   2.89   8.85 
     1   8.81   2.85 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1 
     0   1.59   1.43 
     1   1.40   1.47 

   CPU     0      1 
     0   2.93   2.45 
     1   2.39   2.90 

OK, I got around to spending some more time with this today. I realized that the run_language_modeling.py script can do everything my script was doing, and it uses DDP by default (Note: looking at the most recent version on git, I see that run_language_modeling.py has been replaced by run_clm.py. However, after trying to upgrade transformers to that version, it seems to no longer use the GPU for reasons I don’t have time to debug.).

So now I’m just using that, with:

python -m torch.distributed.launch --nproc_per_node 2 \
    ~/git/transformers/examples/language-modeling/run_language_modeling.py \
    --model_type gpt2 \
    --config_name ./csrc_config \
    --tokenizer_name ./csrc_tokenizer \
    --fp16 --fp16_opt_level O3 \
    --do_train --output_dir csrc_output \
    --per_device_train_batch_size 4 \
    --train_data_file plainsrc_all.txt --block_size 128

For single GPU I drop the torch.distributed.launch and use CUDA_VISIBLE_DEVICES=1, to disable NVLINK I use NCCL_P2P_DISABLE=1 as before. The --block_size 128 argument is to match the default from my training script (without it I run out of GPU RAM).

Results:

So the general observation is that for block size 512, two GPUs without NVLink are about the same performance as a single GPU. For block size 128, two GPUs without NVLink are typically quite a bit slower than a single GPU.

It doesn’t seem like DistributedDataParallel helps with this issue, in other words.

According to this table NV4 means “Connection traversing a bonded set of 4 NVLinks”.

There are some more details in the GA102 whitepaper:

GA102 GPUs utilize NVIDIA’s third-generation NVLink interface, which includes four x4 links, with each link providing 14.0625 GB/sec bandwidth in each direction between two GPUs. Four links provide 56.25 GB/sec bandwidth in each direction, and 112.5 GB/sec total bandwidth between two GPUs. Two RTX 3090 GPUs can be connected together for SLI using NVLink.

OK, so here is my benchmark with the same tool.

edit: my initial benchmark had a bug in it as pointed out by @sgugger as one has to tweak --max_steps if changed to more gpus - I’m proposing to change that and have a way to have a fixed dataset truncation regardless of the number of gpus used. https://github.com/huggingface/transformers/issues/9801

So for 1 gpu, I had to double --max_steps to get the same number of items. The rest of this comment has been updated to reflect the corrected state:

Hardware 2x TITAN RTX 24GB each + NVlink

I get the same bus report w/ and w/o NCCL_P2P_DISABLE=1 - I don’t think nvidia-smi respects this env var:

NCCL_P2P_DISABLE=1 nvidia-smi topo -m

        GPU0    GPU1    CPU Affinity    NUMA Affinity
GPU0     X      NV2     0-23            N/A
GPU1    NV2      X      0-23            N/A

but clearly the runtime is much slower w/o the NVlink as the benchmark shows, so pytorch/cuda does respect it.

Analysis:

1. DP is ~10% slower than DDP w/ NVlink, but ~15% faster than DDP w/o NVlink
2. DDP w/ NVLink doubles the speed of single gpu, so the communication overheard is almost nill in this particular experiment

Here is the full benchmark code and outputs:

# 1 gpu

rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0 python run_clm.py --model_name_or_path gpt2 \
--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --output_dir \
/tmp/test-clm --per_device_train_batch_size 4 --max_steps 400

{'train_runtime': 204.8202, 'train_samples_per_second': 1.953, 'epoch': 0.69}

# DP

rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 python run_clm.py --model_name_or_path gpt2 \
--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --do_train --output_dir \
/tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 110.5948, 'train_samples_per_second': 1.808, 'epoch': 0.69}

# DDP

rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node 2 \
run_clm.py --model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}

# DDP w/o NVlink

rm -r /tmp/test-clm; NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch \
--nproc_per_node 2 run_clm.py --model_name_or_path gpt2 --dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 --do_train --output_dir /tmp/test-clm \
--per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}

I think @sgugger has experience with multi-GPU, and works on the example scripts, pinging him!