aws-sdk-go: [service/s3] high concurrency downloads causes some to fail after a long time

First I am not sure if this is actually an issue but it seems, at least to me, that the it doesn’t run well when there is high concurrency downloads going on and no timeout is set. The exception “connection reset by peer” is thrown at last after a long time (on my box, it took 5~6 minutes for those to fail while the succeeded ones took less than one minute) so looks like the sdk has difficulty on reading the response fast enough in this particular case. I know we can timeout (cancel) the request if it takes too long, by either configuring the sdk with a custom timeout-enabled http client or calling s3.GetObjectWithContext(…) with a timeout Context. But I think 100 concurrent downloads really shouldn’t cause such an issue (in fact I can reproduce it with lower concurrency but 100 guarantees to reproduce it on my box). Also looks like the smaller the network bandwidth is the easier to reproduce the issue.

Version of AWS SDK for Go?

[yizha@yizha-hp-z420 aws-sdk-go]$ echo $GOPATH /home/yizha/work/gows:/home/yizha/work/agency/gows [yizha@yizha-hp-z420 aws-sdk-go]$ pwd /home/yizha/work/gows/src/github.com/aws/aws-sdk-go [yizha@yizha-hp-z420 aws-sdk-go]$ git log HEAD commit ee65a5337cdcef249a6cda35f46c6a14004c1cdf (HEAD -> master, origin/master, origin/HEAD) Author: xibz impactbchang@gmail.com Date: Fri Jan 19 13:54:42 2018 -0800

Only set header tag len if set (#1743)

Version of Go (go version)?

[yizha@yizha-hp-z420 aws-sdk-go]$ go version go version go1.9.2 linux/amd64

What issue did you see?

[yizha@yizha-hp-z420 tmp]$ C_CNT=100 go run test.go concurrency: 100

[1] start downloading … [8] start downloading … [5] start downloading … [99] start downloading … [9] start downloading … [7] start downloading … [0] start downloading … [10] start downloading … [3] start downloading … [4] start downloading … [11] start downloading … [56] start downloading … [14] start downloading … [57] start downloading … [34] start downloading … [16] start downloading … [35] start downloading … [17] start downloading … [36] start downloading … [18] start downloading … [37] start downloading … [58] start downloading … [38] start downloading … [19] start downloading … [59] start downloading … [20] start downloading … [39] start downloading … [60] start downloading … [40] start downloading … [21] start downloading … [61] start downloading … [41] start downloading … [22] start downloading … [62] start downloading … [42] start downloading … [23] start downloading … [63] start downloading … [43] start downloading … [24] start downloading … [64] start downloading … [44] start downloading … [25] start downloading … [65] start downloading … [26] start downloading … [66] start downloading … [45] start downloading … [27] start downloading … [67] start downloading … [68] start downloading … [28] start downloading … [46] start downloading … [30] start downloading … [71] start downloading … [29] start downloading … [47] start downloading … [69] start downloading … [70] start downloading … [49] start downloading … [50] start downloading … [51] start downloading … [31] start downloading … [72] start downloading … [32] start downloading … [12] start downloading … [48] start downloading … [73] start downloading … [53] start downloading … [87] start downloading … [52] start downloading … [54] start downloading … [90] start downloading … [74] start downloading … [88] start downloading … [75] start downloading … [55] start downloading … [77] start downloading … [78] start downloading … [76] start downloading … [79] start downloading … [33] start downloading … [15] start downloading … [80] start downloading … [81] start downloading … [82] start downloading … [85] start downloading … [84] start downloading … [96] start downloading … [97] start downloading … [86] start downloading … [98] start downloading … [94] start downloading … [2] start downloading … [91] start downloading … [92] start downloading … [83] start downloading … [6] start downloading … [13] start downloading … [95] start downloading … [93] start downloading … [89] start downloading … [74] finished download job (read 6198809 bytes) in 43.569521561s [53] finished download job (read 6198809 bytes) in 43.95594173s [4] finished download job (read 6198809 bytes) in 47.707810419s [95] finished download job (read 6198809 bytes) in 48.647493668s [54] finished download job (read 6198809 bytes) in 48.701726836s [46] finished download job (read 6198809 bytes) in 49.470526305s [56] finished download job (read 6198809 bytes) in 49.987636599s [83] finished download job (read 6198809 bytes) in 50.050643396s [13] finished download job (read 6198809 bytes) in 51.928045656s [22] finished download job (read 6198809 bytes) in 53.173156348s [20] finished download job (read 6198809 bytes) in 53.216101938s [55] finished download job (read 6198809 bytes) in 53.600310385s [61] finished download job (read 6198809 bytes) in 53.609563651s [79] finished download job (read 6198809 bytes) in 53.717528561s [94] finished download job (read 6198809 bytes) in 53.954241976s [84] finished download job (read 6198809 bytes) in 54.158461838s [51] finished download job (read 6198809 bytes) in 54.389245387s [43] finished download job (read 6198809 bytes) in 54.611049834s [21] finished download job (read 6198809 bytes) in 54.641270409s [8] finished download job (read 6198809 bytes) in 55.025735593s [39] finished download job (read 6198809 bytes) in 55.026327339s [86] finished download job (read 6198809 bytes) in 55.161876688s [90] finished download job (read 6198809 bytes) in 55.244310288s [85] finished download job (read 6198809 bytes) in 55.396435164s [18] finished download job (read 6198809 bytes) in 55.446654478s [31] finished download job (read 6198809 bytes) in 55.665957191s [82] finished download job (read 6198809 bytes) in 55.790000091s [48] finished download job (read 6198809 bytes) in 55.834006768s [57] finished download job (read 6198809 bytes) in 55.849872747s [70] finished download job (read 6198809 bytes) in 56.149388773s [26] finished download job (read 6198809 bytes) in 56.221490182s [16] finished download job (read 6198809 bytes) in 56.272423036s [67] finished download job (read 6198809 bytes) in 56.44158009s [96] finished download job (read 6198809 bytes) in 56.456063295s [60] finished download job (read 6198809 bytes) in 56.521619324s [50] finished download job (read 6198809 bytes) in 56.771706047s [11] finished download job (read 6198809 bytes) in 56.821399992s [52] finished download job (read 6198809 bytes) in 56.865662218s [66] finished download job (read 6198809 bytes) in 56.97720656s [6] finished download job (read 6198809 bytes) in 56.993605511s [78] finished download job (read 6198809 bytes) in 57.044606204s [30] finished download job (read 6198809 bytes) in 57.167580858s [81] finished download job (read 6198809 bytes) in 57.219458452s [27] finished download job (read 6198809 bytes) in 57.234184741s [80] finished download job (read 6198809 bytes) in 57.294372358s [37] finished download job (read 6198809 bytes) in 57.434658629s [23] finished download job (read 6198809 bytes) in 57.654305373s [91] finished download job (read 6198809 bytes) in 57.924653005s [3] finished download job (read 6198809 bytes) in 57.94704263s [36] finished download job (read 6198809 bytes) in 57.978373205s [45] finished download job (read 6198809 bytes) in 58.207883656s [38] finished download job (read 6198809 bytes) in 58.244114709s [69] finished download job (read 6198809 bytes) in 58.2754416s [41] finished download job (read 6198809 bytes) in 58.293532975s [28] finished download job (read 6198809 bytes) in 58.305003088s [63] finished download job (read 6198809 bytes) in 58.315931935s [32] finished download job (read 6198809 bytes) in 58.320526113s [44] finished download job (read 6198809 bytes) in 58.463426612s [7] finished download job (read 6198809 bytes) in 58.478705522s [92] finished download job (read 6198809 bytes) in 58.620686694s [10] finished download job (read 6198809 bytes) in 58.676208804s [19] finished download job (read 6198809 bytes) in 58.67685684s [49] finished download job (read 6198809 bytes) in 58.70566748s [35] finished download job (read 6198809 bytes) in 58.847224198s [99] finished download job (read 6198809 bytes) in 58.894416924s [76] finished download job (read 6198809 bytes) in 58.968481343s [98] finished download job (read 6198809 bytes) in 58.988596888s [40] finished download job (read 6198809 bytes) in 59.035400909s [58] finished download job (read 6198809 bytes) in 59.043526431s [88] finished download job (read 6198809 bytes) in 59.122832185s [87] finished download job (read 6198809 bytes) in 59.153337061s [24] finished download job (read 6198809 bytes) in 59.182860012s [5] finished download job (read 6198809 bytes) in 59.300469562s [65] finished download job (read 6198809 bytes) in 59.318931113s [68] finished download job (read 6198809 bytes) in 59.327315684s [73] finished download job (read 6198809 bytes) in 59.417350732s [75] finished download job (read 6198809 bytes) in 59.429167583s [89] finished download job (read 6198809 bytes) in 59.441224719s [33] finished download job (read 6198809 bytes) in 59.467230748s [93] finished download job (read 6198809 bytes) in 59.47243084s [62] finished download job (read 6198809 bytes) in 59.571973951s [0] finished download job (read 6198809 bytes) in 59.624225804s [2] finished download job (read 6198809 bytes) in 59.723292307s [64] finished download job (read 6198809 bytes) in 59.759922201s [47] finished download job (read 6198809 bytes) in 59.759636996s [14] finished download job (read 6198809 bytes) in 59.994028361s [59] finished download job (read 6198809 bytes) in 1m0.002918409s [34] finished download job (read 6198809 bytes) in 1m0.070034962s [97] finished download job (read 6198809 bytes) in 1m0.327620044s [77] finished download job (read 6198809 bytes) in 1m0.366530372s [15] finished download job (read 6198809 bytes) in 1m0.402741674s [72] finished download job (read 6198809 bytes) in 1m0.412533226s [29] finished download job (read 6198809 bytes) in 1m0.440142146s [42] finished download job (read 6198809 bytes) in 1m0.510102497s [1] finished download job (read 6198809 bytes) in 1m0.75496559s [71] failed to read body, error: read tcp 10.90.7.237:33446->52.216.64.216:443: read: connection reset by peer [71] finished download job (read 730633 bytes) in 5m45.648985404s [9] failed to read body, error: read tcp 10.90.7.237:33386->52.216.64.216:443: read: connection reset by peer [9] finished download job (read 1409545 bytes) in 5m55.520246483s [12] failed to read body, error: read tcp 10.90.7.237:33566->52.216.64.216:443: read: connection reset by peer [12] finished download job (read 1775113 bytes) in 6m1.201912432s [17] failed to read body, error: read tcp 10.90.7.237:33420->52.216.64.216:443: read: connection reset by peer [17] finished download job (read 748041 bytes) in 6m1.384548141s [25] failed to read body, error: read tcp 10.90.7.237:33516->52.216.64.216:443: read: connection reset by peer [25] finished download job (read 2140681 bytes) in 6m1.941659707s

Steps to reproduce

Run attached example with high concurrency (you will need to modify the access key id and secret key, also change the bucket/key). On my box I can reproduce it with concurrency set to 100 and the s3 object it downloads is about 6MB. You may be able to reproduce it with a different concurrency number, guess it depends on the network bandwidth and the size of the s3 object it downloads.

If you have have an runnable example, please include it.

test.go.txt