multer: File upload sometimes hangs - multer, busboy, or something else?

HI!

I encountered some strange problem while using multer - sometimes it hangs while uploading the file.

When I am testing this behavior on my Ubuntu it succeeds every time, but when I switch to development server it hangs quite often.

I compared the behavior on both environments and added some basic printouts for troubleshooting:

app.use(multer({
  dest: './uploads/',
  rename: function (fieldname, filename) {
    return filename.replace(/\W+/g, '-').toLowerCase();
  },
  onFileUploadStart: function (file) {
    process.stderr.write('Uploading file..........');
  },
  onFileUploadComplete: function (file) {
    process.stderr.write('done\n');
  },

...

}));

On development server Uploading file......... is printed in the console but done doesn’t show up - which suggests it can be multer or busboy issue.

Full code in which I am using multer can be found here: https://github.com/aplikacjespoleczne/WizualizacjaBudzetu

Also for troubleshooting I’ve added the simmilar logging to the multer code (in index.js) :

        fileStream.on('data', function(data) {
          console.log('Next part received!');        // ADDED
          if (data) { file.size += data.length; }
          // trigger "file data" event
          if (options.onFileUploadData) { options.onFileUploadData(file, data); }
        });

And the difference between Ubuntu and development server is the last chunk of data. In both environments Next part received! shows up but on development server done doesn’t - which may suggest that the ‘finish’ event is not emitted.

Can it be multer or busboy? Can you give me some advice where to look further? What else could I trace?

Thanks!

About this issue

Original URL
State: closed
Created 10 years ago
Reactions: 2
Comments: 59 (9 by maintainers)

Most upvoted comments

We faced this issue while trying to upload images from an iOS app to Node.js/Express/Multer via an Nginx reverse proxy. While a majority of the images went through fine, a few specific ones would just not upload. It wasn’t a size issue as well since all images were of similar size and it wasn’t the largest ones which had this problem. It was exactly as @droppedoncaprica has described earlier in this issue - the file was written to the temp location used by Multer but the control was not handed to next.

What finally worked for us was placing all the text fields in the multipart post request before the image file field - i.e. making the image file the last part of the multipart post request. Since then no upload failures. Hope this helps.

Nginx: 1.4.6 Node.js: 0.10.25 Multer: 1.0.0

a9urv on Jun 7, 2016

Just to clarify. In my case the above didn’t work. The solution that did work was adding client_max_body_size to the location portion of the virtual config file for nginx responsible for the proxy settings.

/etc/nginx/conf.d/virtual.conf … server { server_name example.com; location / { client_max_body_size 100m; proxy_pass http://127.0.0.1:3000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; proxy_request_buffering off; } } …

Hope this helps someone, NiKO

nikonovak on Apr 21, 2016

Hello, the problem apparently if in NGINX, after fighting so I found me with error 413 and is a problem of limited upload files to the server, that we must set in NGINX, this would be the way to fix this problem:

open the file:

sudo vim /etc/nginx/nginx.conf

And we look for the next client_max_body_size line and change its value to the size we want (50M). If there is no liena siguiete add the http in the block {…}

client_max_body_size 100M;

and then restart NGINX

sudo service nginx restart

http://atulhost.com/fix-nginx-error-413-request-entity-too-large

kimvex on Mar 7, 2016

+1 first time I’ve used multer – i didnt used cors and credentials and know when both are enabled, that’s a huge problem sometimes it goes well, but only when nodejs restarted and first time it upload small files very well, when i’m trying to make further uploads it hangs up and seems this look like a memory overflow

UPD it works ideally with small files, very small; seems like it is something in nodejs and I suppose that when it will be on production server, there will be also some memory overflows maybe… it depends on a OS, for example I’m using OS X

yevmoroz on Sep 21, 2014

@a9urv may whatever deity you believe in grant you a boon for your bug fix

6 hours of debugging before switching formData order, fook me but it worked!

makeitrein on Sep 10, 2016

Hello, i’m facing the same issue again with the latest of multer version, the problem exactly to what multer does is not handed to next, the files just directly upload to the server but not executed the query process after that, which is sql.query. I’ve looked to @droppedoncaprica example to make custom error but i think i need something scenario as @a9urv does to put image file as the last part of the multipart post request, in the other hand my sql.query need the details from that req.files, what can i solve this problem?

racheats on Jul 13, 2016

So I was stripping down the app I had; some of the code may seem a little odd because of debugging and huge chunks of missing functionality, and sloppiness, but this hung for me. Also, it took about 20 files, then eating dinner, then coming back, then 5 more files or so, then watching the first half of Wolf of Wall sStreet, then coming back to it and at that point it was hung. I was running node in windows powershell but I’ve used the command line as well. (just running like this: PS C:\nodetestingsimple> node server.js). Maybe its just something dumb in my code or “image type” junk. Oh yeah, and fresh npm installs of multer, express and fs just in case I had munged something up.

index.html:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title></title>
</head>
<body>
    <h1> Upload</h1>


    <form method='post' action='/upload' enctype='multipart/form-data'>

        <input type='file' name='image' />
        <input type='submit' />
    </form>


</body>
</html>

and the node script (server.js):

/// <reference path="node_modules/multer/index.js" />
/*Define dependencies.*/

var express = require("express");
var multer = require('multer');
var app = express();
var done = false;
var fs = require('fs');

var fileType = "";
var done = false;

var fileTypeEnum = {
    AUDIO: "audio",
    VIDEO: "video",
    IMAGE: "image",
    DOCUMENT: "document"
};


uploadsuccess = function(fileR) {
    console.log('uploadsuccess');

};

/*Configure the multer.*/

app.use(multer({
    ///////////////////////////////////////
    //dest
    dest: './uploads/',

    ///////////////////////////////////////
    //rename
    rename: function (fieldname, filename) {
        console.log('rename');
        return filename + Date.now();     
    },

    ///////////////////////////////////////
    //In clude empty
    includeEmptyFields: true,

    onError: function (error, next) {
        console.log(error);
        next(error);
    },

        ///////////////////////////////////////
    //Start upload
    onFileUploadStart: function (file, req, res) {
        console.log("start");
        var propValue;
        for (var propName in req) {
            propValue = req[propName];
            console.log(propName, propValue);
        }

        console.log(file.originalname + ' is starting ... onFileUploadStart');
        fileType = "";
        done = false;

        if(file.mimetype == 'image/png' || file.mimetype == 'image/jpg' || file.mimetype == 'image/jpeg' || file.mimetype == 'image/x-tiff'
            || file.mimetype == 'image/tiff' || file.mimetype == 'image/gif' || file.mimetype == 'image/svg+xml') {

            fileType = fileTypeEnum.IMAGE;
        }

        else if(file.mimetype == 'application/x-troff-msvideo'
            || file.mimetype == 'video/avi' || file.mimetype == 'video/msvideo' || file.mimetype == 'video/x-msvideo' || file.mimetype == 'video/x-ms-wmv'
            || file.mimetype == 'video/vnd.uvvu.mp4' ||   file.mimetype == 'video/mp4' || file.mimetype == 'application/mp4'
            || file.mimetype == 'video/mpeg' ||  file.mimetype == 'video/quicktime' ) {
            fileType = fileTypeEnum.VIDEO;
        }

        else if(file.mimetype == 'application/pdf') {
            fileType = fileTypeEnum.DOCUMENT;
        }

        else if (file.mimetype == 'audio/mpeg' || file.mimetype == 'audio/x-mpeg' || file.mimetype == 'audio/mpeg3'
            || file.mimetype == 'audio/x-mpeg-3' || file.mimetype == 'audio/mp4') {
            fileType = fileTypeEnum.AUDIO;
        } else {
            done = true;
            return false;

        }

        console.log(fileType);
        return true;

    },

    ///////////////////////////////////////
    //Upload Complete
    onFileUploadComplete: uploadsuccess,

    ///////////////////////////////////////
    //Parse End
    onParseEnd: function(req, next) {



        console.log("Parse End");


        if (!(req.files.file === undefined)) {

            var resultsObj = { Verified: true, PostID: '1111-1111-22222222-2222' };


                var uploadFilePath = './uploads/' + resultsObj.PostID;

                console.log(req.files);

                fs.rename(req.files.file.path, uploadFilePath, function (er) {


                });



            next();
        }

        next();

    }
}));

/*Handling routes.*/

    app.get('/', function (req, res) {
        res.sendFile(__dirname + "/index.html");
    });

    app.post('/upload', function (req, res) {

        if (done == true) {
            res.end("File uploaded.");
        } else {
            res.end("File uploaded.");
        }
    });

/*Run the server.*/
    app.listen(4000, function () {
        console.log("Working on port 4000");
    });

katachthonios on Jan 12, 2015

This isn’t the best thought-out issue thread, but I am going to chime in here as well.

I’ve been having an issue where file uploads would randomly fail when going through a reverse proxy depending on the file. Larger files will upload fine through the reverse proxy, however certain trouble files will refuse to be received by the server. The request will time out on the client side, and the server never seems to receive anything.

At first I thought it was an nginx issue. If you can upload the file without using the proxy OK and it fails when using the proxy, it must be a reverse proxy issue, right? However the logs from nginx tell a different story.

Looking at the debug logs, the proxy gets the connection and pipes it to the application. On correct downloads, nginx will set up the proxy headers and upstream will respond with it parsing the headers and handling the buffered file. On incorrect downloads, nginx sets the proxied headers and then upstream never responds.

So the issue seems to be with the node application itself. During the investigation, I set up multer like so:

app.use(require('multer')({
  limits: {
    fieldNameSize: 999999999,
    fieldSize: 999999999
  },
  includeEmptyFields: true,
  inMemory: true,
  onFileUploadStart: function(file) {
    console.log('Starting ' + file.fieldname);
  },
  onFileUploadData: function(file, data) {
    console.log('Got a chunk of data!');
  },
  onFileUploadComplete: function(file) {
    console.log('Completed file!');
  },
  onParseStart: function() {
    console.log('Starting to parse request!');
  },
  onParseEnd: function(req, next) {
    console.log('Done parsing!');
    next();
  },
  onError: function(e, next) {
    if (e) {
      console.log(e.stack);
    }
    next();
  }
}));

On correct uploads, logs look something like this:

Starting to parse request!
Starting file
Got a chunk of data!
[ ... ]
Got a chunk of data!
Completed file!
Done parsing!

On bad uploads:

Starting to parse request!
Starting file
Got a chunk of data!
[ ... ]
Got a chunk of data!
Completed file!

On these “bad” files, the parser never finishes parsing the request! Not only that, the onError event never gets caught/ fired either! I’m completely open for this being a reverse proxy misconfiguration, but shouldn’t onError or onParseEnd be fired?

That being said, there isn’t an actual reduced use-case here yet. I’m going to work on getting one put together now.

Edit

Realized onParseEnd needed to have next() called. The issue persists after adding it either way.

I also checked the file sizes and they match from client > nginx file > multer temp file.

Albert-IV on Oct 6, 2014

If you’re using Next.Js and this is happening to you, make sure you have the api bodyParser configured to false (see example here). Hope this saves someone else the time it took me…

Gilthans on Apr 25, 2022