next.js: Updating to next 13.5.2 breaks instrumentation if middlware.ts file exists

Link to the code that reproduces this issue

https://github.com/cabbage-cart/my-minimal-nextjs-issue-reproduction

To Reproduce

run docker-compose --profile test-app up or run locally by npm install then docker-compose up -d && npm run dev

then visit localhost:3000 or localhost:3000/b/[slug]

then visit http://localhost:16686/search, select the test-app service and click Find traces and you’ll see almost every trace is throwing an error Screenshot 2023-09-22 at 2 45 40 am

inspecting the event logs on any trace shows the handleCatchallMiddlewareRequest is throwing an error and not allowing the other spans to come through

event | exception | – | – | – exception.stacktrace | Error at DevServer.handleCatchallMiddlewareRequest (/my-minimal-nextjs-issue-reproduction/node_modules/next/dist/server/next-server.js:1067:33) at async DevServer.handleRequestImpl (/my-minimal-nextjs-issue-reproduction/node_modules/next/dist/server/base-server.js:735:32) | exception.type | Error

Current vs. Expected behavior

Current behaviour @canary

Every trace throwing error

Expected behaviour v13.4.19

traces should not throw error

Verify canary release

I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: darwin
      Arch: x64
      Version: Darwin Kernel Version 22.6.0: Wed Jul  5 22:21:56 PDT 2023; root:xnu-8796.141.3~6/RELEASE_X86_64
    Binaries:
      Node: 20.6.0
      npm: 9.8.1
      Yarn: 1.22.19
      pnpm: N/A
    Relevant Packages:
      next: 13.5.2
      eslint-config-next: N/A
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 5.1.3
    Next.js Config:
      output: N/A

Which area(s) are affected? (Select all that apply)

Middleware / Edge (API routes, runtime)

Additional context

reverting back to 13.4.19. fixes the issue as well as deleting the middleware.ts file.

About this issue

Original URL
State: open
Created 9 months ago
Reactions: 8
Comments: 19

Most upvoted comments

Same here! We are using application insights, and our ‘success’ logs are polluted with Error logs (error: true and resultCode: 200).

We will have to switch back to v13.4.19 for now, or filter out these entries in application insights processor.

stephanbakker on Oct 6, 2023

@cabbage-cart Is this issue now closed because of a work-around/fix in dd-trace? I think this was not per-se related to dd-trace, right? We are using application-insights, so the dd-trace change will not help us.

stephanbakker on Dec 4, 2023

Hey @cabbage-cart, this still seems to be an issue for us. We upgraded to 14.0.0 on our test environment and still pretty much every request gets error response, despite being successful. There is something significantly wrong somewhere.

I looked through our usage of middleware and I cannot see anything out of ordinary in our usage. I know saying that is subjective though.
Would be great to get @ijjk or someone else from next team/vercel to have a look.

re-opening issue

cabbage-cart on Nov 1, 2023

Hey @cabbage-cart, this still seems to be an issue for us. We upgraded to 14.0.0 on our test environment and still pretty much every request gets error response, despite being successful. There is something significantly wrong somewhere.

I looked through our usage of middleware and I cannot see anything out of ordinary in our usage. I know saying that is subjective though.

Would be great to get @ijjk or someone else from next team/vercel to have a look.

szympajka on Nov 1, 2023

Looks like this was related to this https://github.com/vercel/next.js/pull/54813, specifically these lines: https://github.com/vercel/next.js/pull/54813/files#diff-c49c4767e6ed8627e6e1b8f96b141ee13246153f5e9142e1da03450c8e81e96fR1638-R1642 https://github.com/vercel/next.js/blob/a2df451fa8323b4b2eb1e48337243180ac159c69/packages/next/src/server/next-server.ts#L1638-L1642

Not sure I understand why an error is being created here where it wasn’t previously.

david-bell-anz on Sep 26, 2023

It seems that dd-trace 4.18 introduced some fixes to avoid logging that : https://github.com/DataDog/dd-trace-js/releases/tag/v4.18.0

I’m on Next.js 14.0.1 and dd-trace 4.18.0 and I no longer receive those errors.

Thanks! I tried with next.js 14.0.3 and dd-trace 4.21.0, and the error has gone.

gabor-at-reed on Nov 23, 2023

After investigation in my team, we concluded except the noice, this does not create any actual errors for us. It would still be good to remove the cause such error is dispatched though.

@gabor-at-reed, do you have any additional issues caused by it?

@szympajka This issue eats up our DD logs limit in a blink. It’s a serious issue at least financial-wise.

gabor-at-reed on Nov 23, 2023

I can confirm the same. DataDog 200 logs are being polluted with error:

szympajka on Oct 18, 2023

@stephanbakker fair enough. Will leave the issue open still someone from vercel has acknowledged this issues

cabbage-cart on Dec 4, 2023

Something similar happened to me while updating from 13.4.19 to 13.5.3

I get successful DataDog traces wit 200 status code but there is a span with an empty Error.

This is the trace I get
Error
    at NextNodeServer.handleCatchallMiddlewareRequest (/home/node/node_modules/next/dist/server/next-server.js:1067:33)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async NextNodeServer.handleRequestImpl (/home/node/node_modules/next/dist/server/base-server.js:735:32)
APM detected successful requests as errors

same here. Found out after a monitor went off for error tracking.200 responses, but this line just disrupts the rest of the request flow from coming through due to the middleWare flag

https://github.com/vercel/next.js/blob/a2df451fa8323b4b2eb1e48337243180ac159c69/packages/next/src/server/next-server.ts#L1638-L1642

I tried commenting it out in my node_modules and it seemed to fix the issue. Just hoping this gets taken a look at soon.

cabbage-cart on Sep 29, 2023

Something similar happened to me while updating from 13.4.19 to 13.5.3

I get successful DataDog traces wit 200 status code but there is a span with an empty Error.

This is the trace I get

Error
    at NextNodeServer.handleCatchallMiddlewareRequest (/home/node/node_modules/next/dist/server/next-server.js:1067:33)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async NextNodeServer.handleRequestImpl (/home/node/node_modules/next/dist/server/base-server.js:735:32)

APM detected successful requests as errors

oriolgm on Sep 27, 2023

Good point @stephanbakker! Apologies for assuming that it is safe to close this issue now.

szympajka on Dec 6, 2023

Same issue here with the current latest 14.0.3 😭

Also, we can’t downgrade back to 13.4.19 because the sharp package has a high vulnerability issue in npm audit, but we cannot upgrade that without upgrading next.js… but we cannot use packages with high vulnerability issues in production. The sharp is a dependency of the next/image package. Catch of 22.

gabor-at-reed on Nov 22, 2023

The Error creation was created earlier in this commit https://github.com/vercel/next.js/commit/9be1386a253cc55282fb3966738814a5fcb8b631 But it contains more Error creation that looks weird and probably should have some comments explaining why it creates the Error. It looks like Error is used to return the middleware response. Which really looks like there should be a better solution. There should be quite some overhead added by the stack that is attached to an Error in node.

This issue suggests that handleCatchallMiddlewareRequest is called from another place that is not wrapped into the try catch block that is used in handleRoute.

valkum on Sep 26, 2023