graphql-upload: Apollo Server hangs during upload file with graphql-upload
Hello, I am writing to you because I am having issue when I’m using graphql-upload. Let me explain :
I have successfully implemented graphql-upload with the following documentation Enabling file uploads in Apollo Server
As you can see I use Apollo Server (3.5). Here are also the other technologies used in backend :
node(16.18.1)npm(8.19.2)graphql-upload(13.0.0)
And from the frontend, I use apollo-upload-client (17.0.0).
I didn’t really have a problem with setting it up, but it’s when using it that it goes bad. The issue occurs when I upload a file. During the upload, my apollo server is no longer accessible and the frontend no longer works.
What is strange is that the sending is done correctly (without error) but you have to wait between 3 to 6 minutes for a simple 3MB image (and during this time, my application is no longer available).
I specify that the server works locally on my PC and that it is not saturated and also I am the only one to use it, so no memory or cpu overload.
After some research, I find that the problem occurs in the resolver, when doing the stream.pipe(out). To make it simple, I took the code provided in the documentation mentioned above :
singleUpload: async (parent, { file }) => {
const { createReadStream, filename, mimetype, encoding } = await file;
const stream = createReadStream();
const out = require('fs').createWriteStream('local-file-output.txt');
stream.pipe(out);
await finished(out);
return { filename, mimetype, encoding };
},
In my final code I was also thinking of using promisify and waiting for file upload streams, but I’m not sure which is the best way.
Do you know why I have these slownesses? Why is my server hangs? What’s going on? Please, I really need your help on this issue. (sorry for my english, it’s not my mother tongue)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 41 (12 by maintainers)
Hi all.
The maintainer of the graphql-upload-minimal here. I’m going to do a guess why this issue happens.
First of all - we can see that the consumption of the stream stops. Thus server and client hang (timeout).
This is a classic Node issue. Express.js/Connect.js (but not Koa) has it a lot. E.g.:
Voila! Your request hung. Because nobody subscribed to the
"data"event:req.on("data", ...)orreq.pipe(...). https://stackoverflow.com/a/29111941/188475Knowing the above, I was under impression that the
fs-capacitorconsumes the whole request stream regardless, simply because it subscribes to the “data” event of the stream here. Turns out - not always whole.So here is my guess. Node.js or OS (or else) pauses the streaming of data because all the available memory/disk/TCP buffer was consumed and needs draining.
Why it pauses?
Just imagine your Node HTTP server accepted 100 file upload requests simultaneously. Each file is 4.3GB. From my experience the Node.js will easily process all that thanks to streams! At any point of time Node.js holds a little (several kilobytes, say 8KB) buffer for each TCP socket. These 100 upload requests would consume 100*KB=800KB of memory only. So, even if your server has 128MB of RAM, it will still be able to process ONE HUNDRED of 4.3GB files! 😃
However, to make things upload faster you need bigger RAM buffers, the Node.js would use as much memory as the OS allows it to! If Node.js server app has 128MB only then all the 128MB will be used for TCP buffers. After consuming all the RAM the OS will tell the client-side (browsers or curl) to pause sending more data. <- I’m guessing this is why your file upload hangs.
To reproduce the bug above one would need:
@apollo/server/plugin/drainHttpServerfs-capacitoris constrained too) Not 1TB of HDD, but 1GB of free space on your hard drive.Potential bugfix as I see it
The request stream would need to be destroyed if the GraphQL resolver throws or forgets to consume the http request.
However, I have no idea where to put the
req.destroy()line into the module codebase. Sorry.After updating
@apollo/serverfrom v3 to v4, upload requests started “hanging”.I found this little gem in the Apollo documentation:
https://www.apollographql.com/docs/apollo-server/security/cors/#graphql-upload
Apparently the
@apollo/servernow blocks anymultipart/form-datarequests. Adding a special non-empty headerApollo-Require-Preflightfixes the issue for me.You won’t believe me, but I finally found the issue!
As I told you, I develop on a linux VM from my PC through Visual Studio Code. This way when I run my project, Visual Studio Code creates a port forwarding on localhost.
So when I do my tests on the application, I go through localhost which is redirected to my VM from Visual Studio Code and that’s the problem.
Because if I do my tests directly on the IP address of my VM, I don’t have any hangs and the upload is done correctly without freezing the server.
It was not more complicated than that 😃
Sorry to have wasted your time and a big thanks to everyone who took the time to investigate and answer me.
@jaydenseric : You can now close this issue.
@koresar : Thank you very much for your answer, it gives me a little more hope.
@jaydenseric : I switched your example to
@apollo/serverversion 4.3.0, but the issue still the same 😦I will try to open a ticket on the apollo-server side, maybe they will have another approach.
@koresar thanks for the heads-up; in response yesterday I have been doing maintenance preparation work in anticipation of working on this.
Don’t worry about a PR for now; I’ll put a fix together once I fully understand what the problem is. But as deep an explanation as you’re able to share is welcomed!
To explain a bit of context about this:
https://github.com/jaydenseric/graphql-upload/blob/e01b5d5541760d529b06c900883c5fa7febcff00/graphqlUploadExpress.mjs#L54-L63
At the time (I don’t know if browsers have improved in the years since) when a browser
fetchrequest has a server response before the request has finished streaming (a valid thing to do in HTTP), instead of resolving the response instance it would just reject with crappy exceptions. This meant, if for example you had a form registering a user that has a file upload for the avatar, if your GraphQL resolvers throw a validation error about the username being taken while the avatar is still uploading, instead of the GraphQL client being able to cache and render GraphQL errors for the mutation you would just have a fetch error. To prevent this outcome, our GraphQL upload middleware always waits for the request to end before allowing the GraphQL response to be sent. The downside of course is that the user has to wait for the upload to redundantly finish for a failed operation to see the GraphQL errors, but the alternative is they would quickly see no detailed GraphQL errors which is worse.We were not happy having to add these hacky workarounds, really browsers should respect the HTTP specs and not crash a fetch multipart request if there is an early response. We understood it’s a little invasive to monkey patch the Express API to pull it off, but reasoned that if people don’t like that particular behavior it’s trivial for them to import
processRequestand make custom middleware in only a few lines of code with their preferred behavior.I don’t know how constructive this is. But I’m using a fork called graphql-upload-minimal which also seems to have this/ a related issue. Maybe the same thing is happening here, where the server “hangs” because it can’t throw an error? I also tried using this fork but I get the same result. I don’t know but multiple forks having the same issue suddenly might indicate that something else is going on?
I tried turning it into a promise like you said above, but didn’t work for me. I also couldn’t get any logging to work. I’ll try replicating the issue like @NicoSan20 described above.
@jaydenseric : Thank you for your reply.
As soon as I have a little time, I will do the test to switch to V4. I will keep you in touch about the result.