aws-sdk-js: S3 getObject hangs after "write after end" error, when concurrently downloading several files

Using one of the standard Node examples for downloading to a file (http://docs.aws.amazon.com/de_de/sdk-for-javascript/v2/developer-guide/requests-using-stream-objects.html), I am seeing frequent “write after end” errors. Using aws-sdk 2.7.21, Node 7.4.0 and running on Mac OS 10.11.6 – same happens on Ubuntu 16.04. Having searched through the issues here, I notice that there are quite a large number of issues related to getObject, of which some are similar to this one. Most of them are old, and some of them are resolved in the “well, we’ll try this and close this for now …” way. Hence, I can’t tell for sure if this issue duplicates any other issue.

The described happens somewhat randomly, but the below code snippet can be used to reproduce the issue. It seems to happen more often the more concurrent files I am downloading. It also seems to happen more frequently when downloading from a geographically close bucket, i.e. when transfer rate is higher. Surprisingly, the error event is emitted by the read stream – that is, the stream returned by the call to createReadStream(). I am not sure who or what attempts to write to that stream, especially “after end”.

"use strict";

const AWS               = require('aws-sdk');
const createWriteStream = require('fs').createWriteStream;

const s3 = new AWS.S3({region: 'ap-southeast-2', sslEnabled: true, maxRetries: 0});

const download = (key) => {
  console.log(`Starting download of ${key}`);
  const file = createWriteStream(`/tmp/${key}`);
  s3.getObject({Bucket: 'b7f', Key: key, RequestPayer: 'requester'}, (err, data) => {
    if (err) throw new Error(`Unable to download ${key}: ${err}`);
    console.log(`Successfully downloaded ${key}`);
  }).createReadStream().pipe(file);
};

s3.listObjectsV2({Bucket: 'b7f', RequestPayer: 'requester'}, (err, data) => {
  if (err) throw err;
  data.Contents.forEach((key) => download(key.Key));
});

Starting download of 19bsdj5ghpmo008ogck4kg4cc
Starting download of 1ioz372xmx8g48ocsgk0k0gok
Starting download of 1j0lmsd3ucxs0w8g0o0wo800o
Starting download of 1wr9ryp33i74kcgksw4c0c0k8
Starting download of 293etv4hbvms4wok0wwwos8ww
Starting download of 2ffrnrhiymascccogssc4w80g
Starting download of 2kfldsy50xwks4w0ocowkwwww
Starting download of 2vlzsu13aqio4owowwcocwwgc
Starting download of 31p1jpmax3c4k88sk40gs04ow
Starting download of 38m8wp03h50ksooos8gc0skg4
Starting download of 38w1wgi3r8kkkcsk8goooks8o
Starting download of 3dkomp6ijcyswwoo4kw8g4c8
Starting download of 3x2qg6p73u80o4o0gk8cckk4w
Starting download of 41pa7c7mj90kw888cskgkcwow
Starting download of 4e8lq2w1degw8skw4gkcwc440
Starting download of 4nu8l0i0w84kk0ss0s00wkk04
Starting download of 4w0sqjcs0veogcg4wcg8ckc0s
Starting download of 52r4ib76a60wgsco08wscw000
Starting download of 5aboyxyj1n4soo8cks40s800c
Starting download of 5aigz17z3ukgskko8k8kckck0
Starting download of 5ay9kqexokw8s48gccc40ws84
Starting download of 5dmbvh3azdwkggsowwggss0w4
Starting download of 5kieamfse50ckg0w848o48sgc
Starting download of 5szadrkari0wwk00gogoccsk0
Starting download of 64uov4stlt0kgoow0wckccs0g
Starting download of 69sa4mvht6gwgookkgs40occg
Starting download of 6f143zbft484ck4c40wcgosgs
Starting download of 6q8iqzhs9fcww8k4oo8ogcg8c
Starting download of 6sqe4htsh2g4o48os0co88k8k
Starting download of 77lfuttrkocogk4sw8gkw0sgk
Starting download of 78jk7jwdvxsswosowkwcg4gc4
Starting download of 7av7k48y3bc4wkssswocsk4go
Starting download of 7b0py1btnw4c88sg4c8c4o0ow
Starting download of 7sy7brnnhq0wks8gg0o4go8k4
Starting download of 7uwtr3c7gaw44gwk000ww44cg
Starting download of 86fuy0w5o8w00ogk0ccc8kwcg
Starting download of 89bju7u44gsg4ogog8oscw8g8
Starting download of 8kvgfuo4z3sw4kgc8sskowk4w
Starting download of 8nrtaoyymggsgwsgk4808gco0
Starting download of 8wn7ul824oow0sgoocw48ww88
Starting download of 8ymomzzg66g444k0os0o0ww4o
Starting download of 92f7ruaalp0c8wkckw8k08kws
Starting download of 97jd41usu7swgw4gg0g8ok4ok
Starting download of 9inmtfdhxvk0ooc4s48kwoks8
Starting download of 9j53bt7k75kw08www4s0o80sk
Starting download of a160ktt8ivk8s044kk0os8c04
Starting download of a6fdo1a0eio0w0ckk0ssoggk
Starting download of a9p28yg7cw4o4sgc808oc04sk
Starting download of awmos06kc2gcocokk0s8s04wk
Starting download of b3wt9khgcf4gow48w808kokwg
Starting download of bgmdi055xkw0kskckc8cco04o
Starting download of bq8asp4lof4gggksw0koso4kk
Starting download of bsm5bd7bay0o4kw8s0kgcgsgo
Starting download of bt8fcdls4koo4gokwsg40o8wk
Starting download of c0a3bdzmemg4okswck0o0w04k
Starting download of c1zkkek2r6gco4scskowg4kow
Starting download of c2ahmq8usug4gwk08wo4ogw44
Starting download of c58kmj2huzkkk0ss8oo8ksggc
Starting download of c6gc15o0vvkgko0kk4sk088c0
Starting download of c8elt0apo2884kkcw4skgs080
Starting download of cjjsmcrfnzcow0884ggsss44g
Starting download of cph2z1txkhs0s48gcgo8w0cck
Starting download of cqykvxv4pncc40kcwsk84c8s8
Starting download of crjsa26x6aokcw8o80gckc444
Starting download of cwpuc12t8so40cww44s8g0swg
Starting download of d22bjv76y144skg8gs8w4w8k
Starting download of d7howk3t3g0scs000ogogsww8
Starting download of db856jb4so0g0oc84o80s8o8o
Starting download of dn24c19z8gg8s04g4os0gs8ck
Starting download of dvrkmm58ek8w4s8www0cwow0c
Starting download of dvvz1ubleggssksc0wosgoc84
Starting download of dx3t7fubxlsgg888wswcgck4o
Starting download of e09ttmjyci04s8o0wws448ksc
Starting download of e74fd2m3jww80swsww804ocss
Starting download of ece8smc6u5cg8ok8gw4kgokcw
Starting download of eurqa5wo63soks04os4800co0
Starting download of jtskili8fkwkgocscksgcgkg
Starting download of kf4lbkv3qkg4o0wksc804o44
Starting download of oczmcievti8ggc0ggk0ckko0
Successfully downloaded 5aigz17z3ukgskko8k8kckck0
Successfully downloaded 5dmbvh3azdwkggsowwggss0w4
Successfully downloaded 5szadrkari0wwk00gogoccsk0
Successfully downloaded 19bsdj5ghpmo008ogck4kg4cc
Successfully downloaded 4e8lq2w1degw8skw4gkcwc440
Successfully downloaded 4w0sqjcs0veogcg4wcg8ckc0s
Successfully downloaded 41pa7c7mj90kw888cskgkcwow
Successfully downloaded 1wr9ryp33i74kcgksw4c0c0k8
Successfully downloaded 4nu8l0i0w84kk0ss0s00wkk04
Successfully downloaded 2kfldsy50xwks4w0ocowkwwww
Successfully downloaded 38w1wgi3r8kkkcsk8goooks8o
Successfully downloaded 1ioz372xmx8g48ocsgk0k0gok
Successfully downloaded 5kieamfse50ckg0w848o48sgc
Successfully downloaded 13a69rk6ifb40wk8oc4o4wkgk
Successfully downloaded 3x2qg6p73u80o4o0gk8cckk4w
Successfully downloaded 52r4ib76a60wgsco08wscw000
Successfully downloaded 3dkomp6ijcyswwoo4kw8g4c8
Successfully downloaded 293etv4hbvms4wok0wwwos8ww
Successfully downloaded 38m8wp03h50ksooos8gc0skg4
Successfully downloaded 2vlzsu13aqio4owowwcocwwgc
Successfully downloaded 5ay9kqexokw8s48gccc40ws84
Successfully downloaded 2ffrnrhiymascccogssc4w80g
Successfully downloaded 31p1jpmax3c4k88sk40gs04ow
Successfully downloaded 1j0lmsd3ucxs0w8g0o0wo800o
Successfully downloaded 5aboyxyj1n4soo8cks40s800c
Successfully downloaded 64uov4stlt0kgoow0wckccs0g
Successfully downloaded 69sa4mvht6gwgookkgs40occg
Successfully downloaded 6q8iqzhs9fcww8k4oo8ogcg8c
Successfully downloaded 6sqe4htsh2g4o48os0co88k8k
Successfully downloaded 6f143zbft484ck4c40wcgosgs
Successfully downloaded 77lfuttrkocogk4sw8gkw0sgk
Successfully downloaded 7av7k48y3bc4wkssswocsk4go
Successfully downloaded 7uwtr3c7gaw44gwk000ww44cg
Successfully downloaded 78jk7jwdvxsswosowkwcg4gc4
Successfully downloaded 7b0py1btnw4c88sg4c8c4o0ow
Successfully downloaded 7sy7brnnhq0wks8gg0o4go8k4
Successfully downloaded 86fuy0w5o8w00ogk0ccc8kwcg
Successfully downloaded 8wn7ul824oow0sgoocw48ww88
Successfully downloaded 8kvgfuo4z3sw4kgc8sskowk4w
Successfully downloaded 89bju7u44gsg4ogog8oscw8g8
Successfully downloaded 8nrtaoyymggsgwsgk4808gco0
Successfully downloaded 8ymomzzg66g444k0os0o0ww4o
Successfully downloaded 92f7ruaalp0c8wkckw8k08kws
Successfully downloaded 97jd41usu7swgw4gg0g8ok4ok
Successfully downloaded 9inmtfdhxvk0ooc4s48kwoks8
Successfully downloaded a160ktt8ivk8s044kk0os8c04
Successfully downloaded 9j53bt7k75kw08www4s0o80sk
Successfully downloaded a6fdo1a0eio0w0ckk0ssoggk
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: write after end
    at writeAfterEnd (_stream_writable.js:191:12)
    at PassThrough.Writable.write (_stream_writable.js:242:5)
    at PassThrough.ondata (_stream_readable.js:555:20)
    at emitOne (events.js:96:13)
    at PassThrough.emit (events.js:188:7)
    at PassThrough.Readable.read (_stream_readable.js:381:10)
    at flow (_stream_readable.js:761:34)
    at PassThrough.<anonymous> (_stream_readable.js:623:7)
    at emitNone (events.js:91:20)
    at PassThrough.emit (events.js:185:7)

Poking around further and effectively ignoring that error, I noticed that the other transfers would hang. The below code demonstrates that problem. The application hangs forever (I left it a few hours) and only a subset of the files are downloaded; except for the files that failed with the “write after end” error, all the transfers that were in progress at the time that that error happened seem to have hung.

"use strict";

const AWS               = require('aws-sdk');
const createWriteStream = require('fs').createWriteStream;

const s3 = new AWS.S3({region: 'ap-southeast-2', sslEnabled: true, maxRetries: 0});

const status = {};

const download = (key) => {
  console.log(`Starting download of ${key}`);
  status[key]       = 'starting';

  const writeStream = createWriteStream(`/tmp/${key}`);
  const readStream  = s3.getObject({Bucket: 'b7f', Key: key, RequestPayer: 'requester'}, (err, data) => {
    if (err) throw new Error(`Unable to download ${key}: ${err}`);
    console.log(`Successfully downloaded ${key}`);
    status[key] = 'completed';
  }).createReadStream();

  readStream.on('error', (err) => {
    status[key] = `${err}`;
    console.log(`<${key}> Error on read stream: ${err}`);
  });

  readStream.pipe(writeStream);
  readStream.on('data', (chunk) => {
    status[key] = `read ${chunk.byteLength} bytes @ ${Date.now()}`;
  });
  status[key] = 'piping';
};

s3.listObjectsV2({Bucket: 'b7f', RequestPayer: 'requester'}, (err, data) => {
  if (err) throw err;
  data.Contents.sort(() => .5 - Math.random()).forEach((key) => download(key.Key));
});

const printStatus = () => {
  console.log('\n\n=====================================================');
  Object.keys(status).forEach((k) => {
    console.log(`${k}: ${status[k]}`);
  });
};

setInterval(printStatus, 10e3);

In the output I observe that after an error has been reported, the other downloads get stuck.

... thousands of similar lines ...

=====================================================
7uwtr3c7gaw44gwk000ww44cg: completed
jtskili8fkwkgocscksgcgkg: completed
64uov4stlt0kgoow0wckccs0g: completed
d22bjv76y144skg8gs8w4w8k: completed
a6fdo1a0eio0w0ckk0ssoggk: completed
52r4ib76a60wgsco08wscw000: completed
6q8iqzhs9fcww8k4oo8ogcg8c: completed
ece8smc6u5cg8ok8gw4kgokcw: completed
kf4lbkv3qkg4o0wksc804o44: completed
77lfuttrkocogk4sw8gkw0sgk: completed
5aboyxyj1n4soo8cks40s800c: completed
cqykvxv4pncc40kcwsk84c8s8: completed
dx3t7fubxlsgg888wswcgck4o: completed
69sa4mvht6gwgookkgs40occg: completed
e09ttmjyci04s8o0wws448ksc: completed
2kfldsy50xwks4w0ocowkwwww: completed
8kvgfuo4z3sw4kgc8sskowk4w: completed
c58kmj2huzkkk0ss8oo8ksggc: completed
6sqe4htsh2g4o48os0co88k8k: completed
dn24c19z8gg8s04g4os0gs8ck: completed
c6gc15o0vvkgko0kk4sk088c0: completed
5szadrkari0wwk00gogoccsk0: completed
cjjsmcrfnzcow0884ggsss44g: completed
b3wt9khgcf4gow48w808kokwg: completed
crjsa26x6aokcw8o80gckc444: completed
4nu8l0i0w84kk0ss0s00wkk04: completed
oczmcievti8ggc0ggk0ckko0: completed
dvvz1ubleggssksc0wosgoc84: read 9920 bytes @ 1484537172865
c0a3bdzmemg4okswck0o0w04k: completed
9j53bt7k75kw08www4s0o80sk: completed
e74fd2m3jww80swsww804ocss: completed
31p1jpmax3c4k88sk40gs04ow: completed
d7howk3t3g0scs000ogogsww8: completed
dvrkmm58ek8w4s8www0cwow0c: completed
7sy7brnnhq0wks8gg0o4go8k4: completed
9inmtfdhxvk0ooc4s48kwoks8: completed
4e8lq2w1degw8skw4gkcwc440: completed
7av7k48y3bc4wkssswocsk4go: completed
4w0sqjcs0veogcg4wcg8ckc0s: completed
1j0lmsd3ucxs0w8g0o0wo800o: completed
1ioz372xmx8g48ocsgk0k0gok: completed
7b0py1btnw4c88sg4c8c4o0ow: completed
5aigz17z3ukgskko8k8kckck0: completed
bgmdi055xkw0kskckc8cco04o: completed
13a69rk6ifb40wk8oc4o4wkgk: completed
awmos06kc2gcocokk0s8s04wk: read 9920 bytes @ 1484537186655
bq8asp4lof4gggksw0koso4kk: completed
5ay9kqexokw8s48gccc40ws84: completed
bsm5bd7bay0o4kw8s0kgcgsgo: completed
db856jb4so0g0oc84o80s8o8o: completed
1wr9ryp33i74kcgksw4c0c0k8: completed
293etv4hbvms4wok0wwwos8ww: Error: write after end
19bsdj5ghpmo008ogck4kg4cc: piping
3dkomp6ijcyswwoo4kw8g4c8: piping
78jk7jwdvxsswosowkwcg4gc4: piping
a9p28yg7cw4o4sgc808oc04sk: piping
5dmbvh3azdwkggsowwggss0w4: piping
6f143zbft484ck4c40wcgosgs: piping
97jd41usu7swgw4gg0g8ok4ok: piping
8wn7ul824oow0sgoocw48ww88: piping
cph2z1txkhs0s48gcgo8w0cck: piping
eurqa5wo63soks04os4800co0: piping
2vlzsu13aqio4owowwcocwwgc: piping
8ymomzzg66g444k0os0o0ww4o: piping
92f7ruaalp0c8wkckw8k08kws: piping
bt8fcdls4koo4gokwsg40o8wk: piping
c2ahmq8usug4gwk08wo4ogw44: piping
a160ktt8ivk8s044kk0os8c04: piping
41pa7c7mj90kw888cskgkcwow: piping
8nrtaoyymggsgwsgk4808gco0: piping
5kieamfse50ckg0w848o48sgc: piping
cwpuc12t8so40cww44s8g0swg: piping
86fuy0w5o8w00ogk0ccc8kwcg: piping
2ffrnrhiymascccogssc4w80g: piping
89bju7u44gsg4ogog8oscw8g8: piping
c8elt0apo2884kkcw4skgs080: piping
38w1wgi3r8kkkcsk8goooks8o: piping
38m8wp03h50ksooos8gc0skg4: piping
c1zkkek2r6gco4scskowg4kow: piping
3x2qg6p73u80o4o0gk8cckk4w: piping

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 4
  • Comments: 19 (8 by maintainers)

Most upvoted comments

@perhallstroem I did some more testing and it doesn’t appear that the issue is caused by anything the SDK is doing. One work-around would be to pass an agent into your S3 client, then after some time has passed either destroy all the sockets or just the ones that have hung.

We can look into adding this logic into the SDK as well, but we’ll need to come up with a plan on how to do this for the general use case.

Example:

var https = require('https');
var agent = new https.Agent({
    rejectUnauthorized: true,
    maxSockets: 50 // default
});

var AWS = require('aws-sdk');
var s3 = new AWS.S3({
    httpOptions: {
        timeout: 2000,
        agent: agent
    }
});

// sometime when application should be finished
agent.destroy(); // destroys all sockets in use
+3
chrisradek on Feb 27, 2017
Read more comments on GitHub
← aws-sdk-js: RequestTimeout: Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.
aws-sdk-js: S3 listBuckets CORS is not available →