remix: [Bug]: Larger files are not readable via UploadHandler

Which Remix packages are impacted?

remix (Remix core)
create-remix
@remix-run/architect
@remix-run/cloudflare-workers
@remix-run/dev
@remix-run/express
@remix-run/netlify
@remix-run/node
@remix-run/react
@remix-run/serve
@remix-run/server-runtime
@remix-run/vercel

What version of Remix are you using?

1.1.1

What version of Node are you using? Minimum supported version is 14.

16.1.0

Steps to Reproduce

Upload a file using the Form component.

Add a custom UploadHandler:

export const action: ActionFunction = async ({ request }) => {
  const handler: UploadHandler = async ({ name, stream }) => {
    if (name === 'foo') {
      const buff = await new Promise<Buffer>(async (resolve, reject) => {
        const chunks: any = [];

        stream.on("readable", () => {
          let chunk;
          while (null !== (chunk = stream.read())) {
            console.log('chunk...', chunk);
            chunks.push(chunk);
          }
        });

        stream.on("end", () => {
          console.log('end');
          return resolve(Buffer.concat(chunks));
        });
      });
    }
  }

  const form = await unstable_parseMultipartFormData(request, handler);
  // ..
};

Expected Behavior

The Readable stream should trigger the end event once consumed.

Actual Behavior

For small files (e.g. 10kb) the stream is handled correctly, and the end event is called. However, for larger files (90kb fails), a single chunk is logged, but then nothing else happens. The end event is never called.

Am I doing something wrong? I just cannot figure this out. The handling of the stream example is even taken from the nodejs docs.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 21
Comments: 20 (2 by maintainers)

Commits related to this issue

patch-package based on https://github.com/remix-run/remix/issues/1164 — committed to Alazoral/json-parse-example-remix by Alazoral 2 years ago
[BUGFIX] Remove request.clone() in action to make upload "big" files work Linked to issue #1164 It seems that a few people struggle to setup an upload file action It seems that fixing that line se... — committed to arnaudambro/remix by arnaudambro 2 years ago
[BUGFIX] Remove request.clone() in action to make upload "big" files work Linked to issue #1164 It seems that a few people (including myself) struggle to setup an upload file action It seems that ... — committed to arnaudambro/remix by arnaudambro 2 years ago
[BUGFIX] Remove request.clone() in action to make upload "big" files work Linked to issue #1164 It seems that a few people (including myself) struggle to setup an upload file action It seems that ... — committed to arnaudambro/remix by arnaudambro 2 years ago

Most upvoted comments

Hey @aaronksaunders, can you share the process that you used to apply that patch? I’m not used to patching dependencies in the project but I’m actually struggling with the same problem

For now, make the single line change required in node_modules/@remix-run/server-runtime/server.js, then run npx patch-package @remix-run/server-runtime This will create a patch file in patches/xxx Add npx patch-package as a post install script and it will re-apply the change whenever you npm install

+11

jacob-ebey on Dec 27, 2021

I made a mistake in the last comment, it is data.js not server.js @anges244 @jacob-ebey - I just hacked the file for the purpose of the video

line 37 in For now, make the single line change required in node_modules/@remix-run/server-runtime/data.js,

   try {
     result = await action({
-      request: stripDataParam(stripIndexParam(request.clone())),
+      request: stripDataParam(stripIndexParam(request)),
       context: loadContext,
       params: match.params
     });

https://discord.com/channels/770287896669978684/921148525948067850/923329464719536239

aaronksaunders on Jan 5, 2022

I’ll add the comment here because it’s more relevant to this issue:

I’m having issues uploading larger images. After searching on the Discord server, I found this comment from cmwoodall:

I’ve been having the same issue when attempting to upload files… it hangs after the first chunk for files that are greater than one chunk in size. I actually had file uploads working fine in 1.0.6 by backporting the server bits (processMultipartRequest, UploadHandler, etc) and thus figured something else must have changed in the most recent release. After a bit of debugging I think I’ve found the source of the problem – a new request.clone() call that happens with callRouteAction() (https://github.com/remix-run/remix/blob/main/packages/remix-server-runtime/data.ts#L38). When I comment the request.clone() and pass along the original request everything seems to work great. Not my area of expertise but apparently cloning requests in node-fetch can cause issues (https://github.com/node-fetch/node-fetch#custom-highwatermark) and it looks like the highWaterMark is being set earlier in the request but perhaps doesn’t affect subsequent clones 🤷‍♂️

ralrom on Dec 20, 2021

Hey @timonweb if you look at the answers above, you can use patch-package to apply the fix in your other environments

https://github.com/remix-run/remix/issues/1164#issuecomment-1001735640

ralrom on Jan 14, 2022