next.js: Presence of middleware prevents access to raw request bodies greater than or equal to 16,384 bytes (16 KiB)

Building on the workaround found by @Gawdfrey (🙏), if you simply wanted to exclude all api routes from middleware, seems you could do something like this:

export const config = {
  matcher: ['/', '/((?!api/).*)'],
};

I proposed a solution (#41270). Any collaboration or suggestion is welcome.

Just tested on 12.3 and have the same issue

Has anybody tested it with 12.3 yet?

Actually, I just found and attempted with the matcher filter and I was able upload a 295Kb file without deleting the middleware on version 12.2.4! This is of course only an option if you can do without middleware on the paths you exclude.

export const config = {
  runtime: "nodejs",
  matcher: "/about/:path*",
}

A bit late so I will confirm further tomorrow.

Apparently, it worked in v12.0.0 (middleware introduced). I did a bisect to narrow down which version introduced this bug and I can confirm it stopped working in 12.1.1-canary.1. Looking at the version changelog, it’s almost certainly caused by https://github.com/vercel/next.js/pull/34519.

I did some testing and while sending application/json requests the issue is even more pronounced because it’s present even with an empty or very small body. Again, without the middleware requests hits the API endpoint correctly.

Example request:

const response = await fetch("/api/hello", {
    method: "POST",
    headers: {
        "Content-Type": "application/json",
    },
    body: JSON.stringify({}),
});

EDIT: Looks like that small JSON payload issue was resolved by https://github.com/vercel/next.js/pull/35131, but the original issue still persists.

Following up, in my testing the issue occurs regardless of bodyParser setting.

🤖 I’ve updated the repo to 12.2.5-canary.1 and confirmed the issue is still present. (I hope confirming with each canary release is somewhat helpful!)

@jhahn - I believe everyone following this issue appreciates you testing it with every release, thank you. It might be helpful to have a look at the canary release changelog to see whether there were any changes made to the related affected area.

After some additional digging, I’ve realized the multipart/form-data bit is probably a red herring.

I’ve reproduced the issue more precisely with a second test case at https://github.com/jhahn/nextjs-upload-issue/blob/main/pages/large-post.tsx:

/** Add your relevant code here for the issue to reproduce */
export default function Home() {
    const largePost = async () => {
        const formData = new FormData()
        const succeeds = new Uint8Array(16383);
        const fails = new Uint8Array(16384);
        self.crypto.getRandomValues(succeeds);
        self.crypto.getRandomValues(fails);
        const response = await fetch("/api/hello", {
            method: "POST",
            body: fails,
        })
        console.log(await response.json())
    }

    return <button onClick={largePost}>Send Large Post</button>
}

A POST of the succeeds array, with a size of 16,383 bytes, works when middleware is present.

Adding a single byte (the fails array) for a size of 16,384 bytes, causes the POST to hang when middleware is present.

(Again – when there’s no middleware, things work as they should)

Fascinatingly, 16,384 bytes (16 KiB) corresponds exactly to the default highWaterMark option for NodeJS Readable and Writable streams. (https://nodejs.org/api/stream.html#new-streamreadableoptions)

I can confirm we experienced this issue in a production environment, not hosted on Vercel.

This problem appears on the 12.2.3 version. In the 12.2.2 there is no problem. I hope this information may be useful.

I solved this issue above Next@12.2.2 and I can upload large(?) file(up to 16kb)

You can ignore middleware to specific routes through the Matcher

// api route example
/sample-route

// middleware.js or .ts

export const config = {
  matcher: [
    '/((?!sample-route|another-routes).*)', // << Add external routes here
  ],
}

Just tested on 12.3 and have the same issue

Has anybody tested it with 12.3 yet?

@hoersamu @AndyChinViolet I can confirm that we are also having the same issue with 12.3.0. Downgrading to 12.2.0 works.

@filipedeschamps 😱 and 💯 thanks for that very interesting data point!

This was discovered by @aprendendofelipe 🤝

@jhahn thank you for all your reports.

In the meantime, looks like this bug only happens in development and not on Vercels infra. We noticed this with our integration tests, it started to complain for posts above that limit, but in the preview environment, it worked.

I’ve updated the sample repo to 12.2.6-canary.2 and confirmed the issue is still present.

I’ve updated the repo to the final release of 12.2.5 and confirmed the issue is still present.

I also did a bit of additional digging and confirmed the findings of @dmgawel:

• The last official release that doesn’t exhibit the issue described here is 12.1.0
• The last pre-release that doesn’t exhibit the issue is 12.1.1-canary.0

If I had to hazard a guess, https://github.com/vercel/next.js/pull/34519 is the PR that introduced the issue. https://github.com/vercel/next.js/issues/34966 looks relevant, as does its fix https://github.com/vercel/next.js/pull/35131.

Great find @Pamboli, looks like the issue was first introduced in 12.1.1-canary.1, then fixed, and then a similar bug was reintroduced in 12.2.3-canary.17 (see new commits comparing to canary 16).

@filipedeschamps 😱 and 💯 thanks for that very interesting data point!

We’re not hosting on Vercel, but we didn’t even think to try what we assumed was non-functioning code in our production environment. We’ll do some additional testing on this end and update the issue with our findings.

I’ve updated the sample repo to 12.2.6-canary.1 and confirmed this issue is still present.

I can also confirm @Gawdfrey’s “matcher filter” workaround solves the issue, assuming you don’t need middleware on the excluded path(s).

@Gawdfrey FWIW, the workaround proposed by @Pamboli (downgrading to 12.2.2) did not work for us, either.

This problem appears on the 12.2.3 version. In the 12.2.2 there is no problem. I hope this information may be useful.

Verified on my end and working when I changed the version to 12.2.2

Thank you @Pamboli !

🤖 I’ve updated the repo to 12.2.5-canary.1 and confirmed the issue is still present. (I hope confirming with each canary release is somewhat helpful!)

I’ve updated the repo to 12.2.5-canary.0 and confirmed the issue is still present.

I’ve updated the repo to 12.2.4-canary.12 and confirmed the issue is still present.