azure-pipelines-tasks: CopyFilesOverSSHV0 unbearably slow

Type: Bug

Enter Task Name: CopyFilesOverSSHV0

Environment

• Server - Azure Pipelines

  • If using Azure Pipelines, provide the account name, team project name, build definition name/build number:
    • Account Name: numberjak
    • Team Project: Magpie
    • Release Definition Name: DigitalOcean
    • Release Name: Release-4

• Agent - Hosted or Private:

  • If using Hosted agent, provide agent queue name: WIN-DF00C5S5J33 (this is the machine name. hope that’s what you mean)

Issue Description

My release pipeline copies build artifacts to a remote server on DigitalOcean using the CopyFilesOverSSHV0 task. The files are around 500MB total in size. However the process of copying these files to the remote server consistently takes around 43 minutes, which is obviously ridiculously slow.

Running some benchmarks on the remote server the files are being sent to, I can see the problem is not on DigitalOcean server:

I/O Benchmarks:

root@blahblah:~# dd if=/dev/zero of=benchmark bs=1M count=2048 conv=fdatasync
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 2.83064 s, 759 MB/s
root@blahblah:~# dd if=/dev/zero of=benchmark bs=64K count=32K conv=fdatasync
32768+0 records in
32768+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 3.3467 s, 642 MB/s

Download speed benchmark:

root@blahblah:~# wget --output-document=/dev/null http://speedtest.wdc01.softlayer.com/downloads/test500.zip
--2020-06-19 16:15:09--  http://speedtest.wdc01.softlayer.com/downloads/test500.zip
Resolving speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)... 158.85.230.20, 2607:f0d0:3006:6c::4
Connecting to speedtest.wdc01.softlayer.com (speedtest.wdc01.softlayer.com)|158.85.230.20|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 524288000 (500M) [application/zip]
Saving to: ‘/dev/null’

/dev/null                                         100%[=============================================================================================================>] 500.00M  46.6MB/s    in 30s     

2020-06-19 16:15:48 (16.7 MB/s) - ‘/dev/null’ saved [524288000/524288000]

Task logs

ReleaseLogs_5.zip