node-ftp: Error after downloading lots of files
I have a timer that runs every 100ms. It goes through a list of known remote file paths on the FTP server and attempts to download them to the local machine.
The timer has locks to ensure that only 3 requests are active at any one time (every time a download finishes it releases another slot to the pool).
This works well for about 20 seconds before get()
errors out with
Unable to make data connection
Here is a reduced example of what is happening
function ftpController() {
this.conn = new ftpClient();
}
ftpController.prototype.downloadFile = function(remotePath, cb) {
cb = cb || function() {};
localPath = mods.folderPath + remotePath;
ftp.conn.get(remotePath, function(err, stream) {
if (err) {
debug('** Error downloading ' + remotePath, err);
cb(false);
} else {
stream.pipe(fs.createWriteStream(localPath));
cb(true);
}
});
};
// Stripped code example
ftp.downloadFile('/test/test1.txt', releaseAnotherDownload);
ftp.downloadFile('/test/test2.txt', releaseAnotherDownload);
ftp.downloadFile('/test/test3.txt', releaseAnotherDownload);
Should I be creating a new connection for each download rather than trying to use the same, is my FTP server throttling me because I’m not making a new connection for each file?
About this issue
- Original URL
- State: open
- Created 9 years ago
- Reactions: 2
- Comments: 15
I was just bitten by this as well. It looks like the problem is in
connection.js
withself._pasvSocket = socket;
. It replaces _pasvSocket each time. So for example, GET 1 sets _pasvSocket, GET 2 overrides _pasvSocket, then GET 1 completes, which clears _pasvSocket, so GET 2 fails withUnable to make data connection
. In my testing, it was always the third GET that was causing the problem, but I would guess that it depends on the timing for each GET.+1
@vintuwei I did not. I ended up enforcing one file at a time via
async.series
. You can see my code at https://github.com/phillipgreenii/node-ftp-stream/blob/master/src/index.js