next.js: Presence of middleware prevents access to raw request bodies greater than or equal to 16,384 bytes (16 KiB)
Verify canary release
- I verified that the issue exists in the latest Next.js canary release
Provide environment information
Operating System:
Platform: darwin
Arch: arm64
Version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
Binaries:
Node: 18.7.0
npm: 8.15.0
Yarn: 1.22.19
pnpm: 7.8.0
Relevant packages:
next: 12.2.4-canary.9
eslint-config-next: N/A
react: 18.2.0
react-dom: 18.2.0
What browser are you using? (if relevant)
N/A
How are you deploying your application? (if relevant)
N/A
Describe the Bug
When attempting to upload a file over a few kilobytes (e.g. sending a POST
request with a binary body and a Content-Type
of multipart/form-data
) via fetch
or curl
, the request stalls, then fails with the error:
error - Error: aborted
at connResetException (node:internal/errors:704:14)
at abortIncoming (node:_http_server:700:17)
at socketOnClose (node:_http_server:694:3)
at Socket.emit (node:events:525:35)
at TCP.<anonymous> (node:net:757:14) {
middleware: true
}
This occurs only for API pages with…
export const config = {
api: {
bodyParser: {
bodyParser: false,
},
},
}
…and only when middleware is present; even something as basic as:
import {NextRequest, NextResponse} from "next/server"
export async function middleware(req: NextRequest) {
return NextResponse.next()
}
Removing the middleware fixes the issue. Of note, very small request bodies (e.g. < 1kb files) work even in the presence of middleware.
Expected Behavior
Sending a POST request to an API endpoint with a Content-Type of multipart/form-data
along with a reasonably sized (~200kB) binary payload should work and not stall.
Link to reproduction
https://github.com/jhahn/nextjs-upload-issue
To Reproduce
pages/index.tsx
import type { NextPage } from 'next'
import Head from 'next/head'
import Image from 'next/image'
import styles from '../styles/Home.module.css'
const Home: NextPage = () => {
const uploadFile = async (files: FileList | null) => {
if (!files) return
const formData = new FormData()
formData.append("file", files[0])
const response = await fetch("/api/hello", {
method: "POST",
body: formData,
})
console.log(await response.json())
}
return <input type="file" onChange={(e) => uploadFile(e.target.files)} />
}
export default Home
pages/api/hello.ts
:
import type { Readable } from 'node:stream';
import type { NextApiRequest, NextApiResponse } from 'next'
export const config = {
api: {
bodyParser: {
bodyParser: false,
},
},
}
async function buffer(readable: Readable) {
const chunks = [];
for await (const chunk of readable) {
chunks.push(typeof chunk === 'string' ? Buffer.from(chunk) : chunk);
}
return Buffer.concat(chunks);
}
export default async function (req: NextApiRequest, res: NextApiResponse) {
if (req.method === 'POST') {
const buf = await buffer(req.body);
const rawBody = buf.toString('utf8');
// Can do something here...
res.json({ rawBody });
} else {
res.setHeader('Allow', 'POST');
res.status(405).end('Method Not Allowed');
}
}
The code for pages/api/hello.ts
was adapted from https://vercel.com/support/articles/how-do-i-get-the-raw-body-of-a-serverless-function. However, I had to change const buf = await buffer(req);
to const buf = await buffer(req.body);
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 21
- Comments: 44 (5 by maintainers)
Commits related to this issue
- fix(stream): Allows body larger than 16 KiB with middleware (#41270) Fixes #39262 The solution is to call `stream.push(null)` to trigger the `end` event which allows `getRawBody` to run completel... — committed to vercel/next.js by aprendendofelipe 2 years ago
- fix(stream): Allows body larger than 16 KiB with middleware (#41270) Fixes #39262 The solution is to call `stream.push(null)` to trigger the `end` event which allows `getRawBody` to run completel... — committed to Kikobeats/next.js by aprendendofelipe 2 years ago
Building on the workaround found by @Gawdfrey (🙏), if you simply wanted to exclude all api routes from middleware, seems you could do something like this:
I proposed a solution (#41270). Any collaboration or suggestion is welcome.
Just tested on 12.3 and have the same issue
Actually, I just found and attempted with the matcher filter and I was able upload a 295Kb file without deleting the middleware on version 12.2.4! This is of course only an option if you can do without middleware on the paths you exclude.
A bit late so I will confirm further tomorrow.
Apparently, it worked in
v12.0.0
(middleware introduced). I did a bisect to narrow down which version introduced this bug and I can confirm it stopped working in12.1.1-canary.1
. Looking at the version changelog, it’s almost certainly caused by https://github.com/vercel/next.js/pull/34519.I did some testing and while sending
application/json
requests the issue is even more pronounced because it’s present even with an empty or very small body. Again, without the middleware requests hits the API endpoint correctly.Example request:
EDIT: Looks like that small JSON payload issue was resolved by https://github.com/vercel/next.js/pull/35131, but the original issue still persists.
Following up, in my testing the issue occurs regardless of
bodyParser
setting.@jhahn - I believe everyone following this issue appreciates you testing it with every release, thank you. It might be helpful to have a look at the canary release changelog to see whether there were any changes made to the related affected area.
After some additional digging, I’ve realized the
multipart/form-data
bit is probably a red herring.I’ve reproduced the issue more precisely with a second test case at https://github.com/jhahn/nextjs-upload-issue/blob/main/pages/large-post.tsx:
A
POST
of thesucceeds
array, with a size of 16,383 bytes, works when middleware is present.Adding a single byte (the
fails
array) for a size of 16,384 bytes, causes thePOST
to hang when middleware is present.(Again – when there’s no middleware, things work as they should)
Fascinatingly, 16,384 bytes (16 KiB) corresponds exactly to the default
highWaterMark
option for NodeJS Readable and Writable streams. (https://nodejs.org/api/stream.html#new-streamreadableoptions)I can confirm we experienced this issue in a production environment, not hosted on Vercel.
This problem appears on the 12.2.3 version. In the 12.2.2 there is no problem. I hope this information may be useful.
I solved this issue above Next@12.2.2 and I can upload large(?) file(up to 16kb)
You can ignore middleware to specific routes through the Matcher
@hoersamu @AndyChinViolet I can confirm that we are also having the same issue with 12.3.0. Downgrading to 12.2.0 works.
This was discovered by @aprendendofelipe 🤝
@jhahn thank you for all your reports.
In the meantime, looks like this bug only happens in
development
and not on Vercels infra. We noticed this with our integration tests, it started to complain for posts above that limit, but in thepreview
environment, it worked.I’ve updated the sample repo to
12.2.6-canary.2
and confirmed the issue is still present.I’ve updated the repo to the final release of
12.2.5
and confirmed the issue is still present.I also did a bit of additional digging and confirmed the findings of @dmgawel:
12.1.0
12.1.1-canary.0
If I had to hazard a guess, https://github.com/vercel/next.js/pull/34519 is the PR that introduced the issue. https://github.com/vercel/next.js/issues/34966 looks relevant, as does its fix https://github.com/vercel/next.js/pull/35131.
Great find @Pamboli, looks like the issue was first introduced in
12.1.1-canary.1
, then fixed, and then a similar bug was reintroduced in12.2.3-canary.17
(see new commits comparing to canary 16).@filipedeschamps 😱 and 💯 thanks for that very interesting data point!
We’re not hosting on Vercel, but we didn’t even think to try what we assumed was non-functioning code in our production environment. We’ll do some additional testing on this end and update the issue with our findings.
I’ve updated the sample repo to
12.2.6-canary.1
and confirmed this issue is still present.I can also confirm @Gawdfrey’s “matcher filter” workaround solves the issue, assuming you don’t need middleware on the excluded path(s).
@Gawdfrey FWIW, the workaround proposed by @Pamboli (downgrading to
12.2.2
) did not work for us, either.Verified on my end and working when I changed the version to 12.2.2
Thank you @Pamboli !
🤖 I’ve updated the repo to
12.2.5-canary.1
and confirmed the issue is still present. (I hope confirming with each canary release is somewhat helpful!)I’ve updated the repo to
12.2.5-canary.0
and confirmed the issue is still present.I’ve updated the repo to
12.2.4-canary.12
and confirmed the issue is still present.