next.js: Building production when using localhost:3000 in a fetch gives ECONNREFUSED

Link to the code that reproduces this issue

https://github.com/douwepausma/build-with-routehandlers-reproduction

To Reproduce

1. Create new next-app npx create-next-app@latest
2. Create a simple route.

// /app/api/model/route.js
export async function GET() {
    return Response.json({ data: {message: 'Hello World!'} })
}

1. Fetch route from page.

// /app/page.js
export default async function Home() {
  try {
    const res = await fetch('http://localhost:3000/api/model');
  } catch (error) {
    console.log(error);
  }
  
  return null;
}

1. npm run build

Current vs. Expected behavior

Current behavior

TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11372:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error: connect ECONNREFUSED ::1:3000
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -61,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 3000
  }
}

Expected behavior Successful build

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.0.0: Fri Sep 15 14:43:05 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6020
Binaries:
  Node: 18.18.2
  npm: 9.8.1
  Yarn: 1.22.19
  pnpm: 8.11.0
Relevant Packages:
  next: 14.0.4-canary.32
  eslint-config-next: N/A
  react: 18.2.0
  react-dom: 18.2.0
  typescript: N/A
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Data fetching (gS(S)P, getInitialProps), Middleware / Edge (API routes, runtime)

Additional context

What I’ve tried

1. Use 127.0.0.1 instead of localhost
2. Adding export const runtime = 'edge' to page fixed build issue, but breaks app later on (when deploying on vercel).*
3. Adding const dynamic = 'force-dynamic' to page fixed build issue, but breaks app later on (when deploying on vercel).*

*These fixes shouldn’t be necessary as I’ve successfully used route handlers in the past without it.