next.js: Dynamic Router - url decoded parameter results in javascript 404

Verify canary release

  • I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
      Platform: darwin
      Arch: x64
      Version: Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64
    Binaries:
      Node: 18.15.0
      npm: 9.5.0
      Yarn: 1.22.19
      pnpm: N/A
    Relevant Packages:
      next: 13.4.16
      eslint-config-next: 13.4.16
      react: 18.2.0
      react-dom: 18.2.0
      typescript: 5.1.6
    Next.js Config:
      output: N/A

Which area(s) of Next.js are affected? (leave empty if unsure)

Routing (next/router, next/navigation, next/link)

Link to the code that reproduces this issue or a replay of the bug

https://github.com/klassm/next-url-encoding-reproduce

To Reproduce

  • Checkout the reproduction repo
  • Run the example
npm i
npm run startReproduce # builds the next app in prod mode, starts the dev server
npm run testReproduce # runs jest

One of the two tests will fail:

> testReproduce
> jest

 FAIL  ./runMe.spec.ts
  run me
    ✓ should be able to access url encoded chunks (113 ms)
    ✕ should be able to access non url encoded chunks (25 ms)

  ● run me › should be able to access non url encoded chunks

    AxiosError: Request failed with status code 404

      at settle (node_modules/axios/lib/core/settle.js:19:12)
      at Unzip.handleStreamEnd (node_modules/axios/lib/adapters/http.js:570:11)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 passed, 2 total
Snapshots:   0 total
Time:        2.541 s

Describe the Bug

When opening a dynamic routed page, next.js will load a js file from the server. The dynamic parameter, in this case [id], will be url encoded. According to RFC2616 this parameter can be url encoded or not - the app should handle it the same way [1]. However, when trying to load the js file without url encoding the server returns a 404.

Why is this bad?

It looks like using nginx as a reverse proxy in front of a next application will already remove the url encoding from the frontend. The request then runs in a 404, resulting in the page not properly being loaded.

I also looked when this issue was introduced - with 13.4.12 it was still green.

[1]

Characters other than those in the "reserved" and "unsafe" sets (see
   RFC 2396 [42]) are equivalent to their ""%" HEX HEX" encoding.

   For example, the following three URIs are equivalent:

      [http://abc.com:80/~smith/home.html](http://abc.com/~smith/home.html)
      [http://ABC.com/%7Esmith/home.html](http://abc.com/~smith/home.html)
      http://abc.com/:/%7esmith/home.html

Expected Behavior

Both URL encoded and non URL encoded URLs are handled the same way and result in a 200.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Reactions: 11
  • Comments: 17 (4 by maintainers)

Most upvoted comments

This bug also applies when deploying to an IIS web server using iisnode as an intermediary. Running the same app via “npm run start” without iisnode makes the error go away. Presumably because as the OP indicates, the brackets are getting needlessly URL decoded in the middle.

I agree that the resolution needs to be Next.js’s production router accepting either encoded or decoded brackets in URLs. But I found an interim solution that works in the iisnode case at least. The solution involved editing the server.js file iisnode is configured to bootstrap with. The solution replaces any brackets found in req.url with their URL-encoded equivalents. This is a laser-focused hack for dealing specifically with this problem and not others that may arise from URL transformation like this. Partly because I want to minimize URL hacking security risks. Partly because this should be a short-term patch and not the final solution. Worked for us though. Here’s our server.js file:

const { createServer } = require('http');
const { parse } = require('url');
const next = require('next');

const dev = process.env.NODE_ENV !== 'production';
const hostname = 'localhost';
const port = process.env.PORT || 3000;

// when using middleware `hostname` and `port` must be provided below
const app = next({ dev, hostname, port });
const handle = app.getRequestHandler();

app.prepare().then(() => {
    createServer(async (req, res) => {

        // Emergent bug in Next.js: doesn't deal wll with brackets in 
        // URL when iisnode or other reverse proxy is in the middle.
        // See https://github.com/vercel/next.js/issues/54325
        let reqUrl = req.url
            .replace(/\[/g, '%5B')
            .replace(/\]/g, '%5D');
        //console.log(`>>> ` + reqUrl);

        try {
            // Be sure to pass `true` as the second argument to `url.parse`.
            // This tells it to parse the query portion of the URL.
            const parsedUrl = parse(reqUrl, true);

            await handle(req, res, parsedUrl);

        } catch (err) {
            console.error('Error occurred handling', reqUrl, err);
            res.statusCode = 500;
            res.end('internal server error');
        }
    })
        .listen(port, (err) => {
            if (err) throw err;
            console.log(`> Ready on http://${hostname}:${port}`);
        });
});

Same problem here on production, when using dynamic routes. I downgraded to 13.4.12 now.

@navivanuva that version is outdated, please upgrade to latest version 13.5.4

Anyone able to test this PR for iisnode too? If we can confirm it fixes this scenario as well as nginx, then I can link both issues to the PR as a fix.

#56187

This bug also applies when deploying to an IIS web server using iisnode as an intermediary. Running the same app via “npm run start” without iisnode makes the error go away. Presumably because as the OP indicates, the brackets are getting needlessly URL decoded in the middle.

I agree that the resolution needs to be Next.js’s production router accepting either encoded or decoded brackets in URLs. But I found an interim solution that works in the iisnode case at least. The solution involved editing the server.js file iisnode is configured to bootstrap with. The solution replaces any brackets found in req.url with their URL-encoded equivalents. This is a laser-focused hack for dealing specifically with this problem and not others that may arise from URL transformation like this. Partly because I want to minimize URL hacking security risks. Partly because this should be a short-term patch and not the final solution. Worked for us though. Here’s our server.js file:

const { createServer } = require('http');
const { parse } = require('url');
const next = require('next');

const dev = process.env.NODE_ENV !== 'production';
const hostname = 'localhost';
const port = process.env.PORT || 3000;

// when using middleware `hostname` and `port` must be provided below
const app = next({ dev, hostname, port });
const handle = app.getRequestHandler();

app.prepare().then(() => {
    createServer(async (req, res) => {

        // Emergent bug in Next.js: doesn't deal wll with brackets in 
        // URL when iisnode or other reverse proxy is in the middle.
        // See https://github.com/vercel/next.js/issues/54325
        let reqUrl = req.url
            .replace(/\[/g, '%5B')
            .replace(/\]/g, '%5D');
        //console.log(`>>> ` + reqUrl);

        try {
            // Be sure to pass `true` as the second argument to `url.parse`.
            // This tells it to parse the query portion of the URL.
            const parsedUrl = parse(reqUrl, true);

            await handle(req, res, parsedUrl);

        } catch (err) {
            console.error('Error occurred handling', reqUrl, err);
            res.statusCode = 500;
            res.end('internal server error');
        }
    })
        .listen(port, (err) => {
            if (err) throw err;
            console.log(`> Ready on http://${hostname}:${port}`);
        });
});

This solves a bug that I’ve been struggling with for several weeks and therefore couldn’t update. Regardless, I hope the Next.js team fixes the bug. But in any case a great fallback, thank you very much.

We also stumbled upon this today. reverting to 13.4.12 for now…

Same with nginx 1.25.2 (reverse proxy) & Next.js containing dynamic routes. However, if I start the server locally with npm run build followed by npm run start, there are no problems to report. It’s very strange

Downgrade to 13.4.12 solve the issue 😕