next.js: `next dev` will not resolve missing routes using yarn 2 zero-install

This seems to be related to Plug’n’Play - setting nodeLinker: node-modules in .yarnrc.yml causes error pages to appear again. Similarly, running yarn unplug next makes the issue disappear, since the next module is unpacked into .yarn/unplugged.

I tried to do some debugging to determine the cause and was unable to find it, but here are some findings, hopefully they are helpful.

In on-demand-entry-helper.ts, the ensurePage function is called for the 404 page, and it returns a promise that is intended to resolve when the next/dist/pages/_error page is compiled, but that promise never resolves. This causes the behavior experienced where 404’s never appear in the browser, the tab just spins forever.

Specifically, this code:

https://github.com/vercel/next.js/blob/5fff814ca1e7f78b50535a55d6f99bd842c4e049/packages/next/server/on-demand-entry-handler.ts#L203-L210

That registers a callback for the normalized page '/next/dist/pages/_error' and requests that the invalidator invalidate compilation. The call to invalidator.invalidate() does trigger some kind of recompile, reaching the code here that is supposed to emit the event for compile success:

https://github.com/vercel/next.js/blob/bddb02286fdcfb83fc97fa91df378cb6d92778e8/packages/next/server/on-demand-entry-handler.ts#L71-L94

• With PnP, the entries variable is an empty object ({}), and pagePaths does not contain '/next/dist/pages/_error', so no event is emitted, the promise never resolves, and the browser hangs.
• Without PnP, the entries variable is an object with a key for '/next/dist/pages/_error', and pagePaths contains '/next/dist/pages/_error', so an event is emitted and the 404 page is displayed.

So, it would appear that the entry for next/dist/pages/_error is not getting registered with Webpack, or the hot reloader, or some other middleware in a way that works with Yarn PnP. I did some brief investigation into how this works currently, but don’t understand enough to determine the cause.

Perhaps related is the next-client-pages-loader, it receives next/dist/pages/_error as an “absolutePagePath”, and it’s not clear if it would correctly use require.resolve or similar to resolve the page path with PnP, but some attempts to resolve the path prior with require.resolve caused Yarn to start complaining about importing private-next-pages, which is supposed to be resolved by Webpack, so I clearly do not fully understand the loader structure.

Hopefully that’s helpful and someone can locate the root cause!

+1 to fixing this. I worked around this by re-exporting the error page:

// pages/_error.tsx
export { default } from 'next/error'

EDIT: Actually, just doing this gives a warning

Warning: You have added a custom /_error page without a custom /404 page. This prevents the 404 page from being auto statically optimized.
See here for info: https://nextjs.org/docs/messages/custom-error-no-custom-404

So instead, I just added my own pages/404.tsx page.

I think I’ve tracked down the cause of the problem. Thank you for your research @jacobwgillespie.

The hot reloader appears to determine whether a page exists by checking if write access is granted.

https://github.com/vercel/next.js/blob/76e2bb57adfd1dae53da912c5c666e63443dacf7/packages/next/server/hot-reloader.ts#L324-L329

https://github.com/vercel/next.js/blob/76e2bb57adfd1dae53da912c5c666e63443dacf7/packages/next/build/is-writeable.ts#L1-L10

I suspect that because the built in Next.js pages are contained within zip files, the virtual file system provided by Yarn PnP reports the file is read only (write access declined).

Changing the line of code in hot-reloader.ts to something like this appears to work:

const pageExists =
  serverBundlePath.startsWith("pages/next/dist/pages") ||
  (await isWriteable(absolutePagePath))

To test this yourself, you can use the Yarn 2 patch feature: https://yarnpkg.com/cli/pack https://yarnpkg.com/features/protocols#patch

next-patch.diff:

diff --git a/dist/server/hot-reloader.js b/dist/server/hot-reloader.js
index 5154c7529a44788bf909c97cde5c34c9b0905bbc..e931fee355a4b35f5137a3810b7ce10d33172dac 100644
--- a/dist/server/hot-reloader.js
+++ b/dist/server/hot-reloader.js
@@ -12,7 +12,7 @@ const{preflight}=addCorsSupport(req,res);if(preflight){return{};}// When a reque
 // by adding the page to on-demand-entries, waiting till it's done
 // and then the bundle will be served like usual by the actual route in server/index.js
 const handlePageBundleRequest=async(pageBundleRes,parsedPageBundleUrl)=>{const{pathname}=parsedPageBundleUrl;const params=matchNextPageBundleRequest(pathname);if(!params){return{};}let decodedPagePath;try{decodedPagePath=`/${params.path.map(param=>decodeURIComponent(param)).join('/')}`;}catch(_){const err=new Error('failed to decode param');err.code='DECODE_FAILED';throw err;}const page=(0,_normalizePagePath.denormalizePagePath)(decodedPagePath);if(page==='/_error'||_constants2.BLOCKED_PAGES.indexOf(page)===-1){try{await this.ensurePage(page);}catch(error){await renderScriptError(pageBundleRes,error);return{finished:true};}const errors=await this.getCompilationErrors(page);if(errors.length>0){await renderScriptError(pageBundleRes,errors[0],{verbose:false});return{finished:true};}}return{};};const{finished}=await handlePageBundleRequest(res,parsedUrl);for(const fn of this.middlewares){await new Promise((resolve,reject)=>{fn(req,res,err=>{if(err)return reject(err);resolve();});});}return{finished};}async clean(){return(0,_recursiveDelete.recursiveDelete)((0,_path.join)(this.dir,this.config.distDir),/^cache/);}async getWebpackConfig(){const pagePaths=await Promise.all([(0,_findPageFile.findPageFile)(this.pagesDir,'/_app',this.config.pageExtensions),(0,_findPageFile.findPageFile)(this.pagesDir,'/_document',this.config.pageExtensions)]);const pages=(0,_entries.createPagesMapping)(pagePaths.filter(i=>i!==null),this.config.pageExtensions);const entrypoints=(0,_entries.createEntrypoints)(pages,'server',this.buildId,this.previewProps,this.config,[]);return Promise.all([(0,_webpackConfig.default)(this.dir,{dev:true,isServer:false,config:this.config,buildId:this.buildId,pagesDir:this.pagesDir,rewrites:this.rewrites,entrypoints:entrypoints.client}),(0,_webpackConfig.default)(this.dir,{dev:true,isServer:true,config:this.config,buildId:this.buildId,pagesDir:this.pagesDir,rewrites:this.rewrites,entrypoints:entrypoints.server})]);}async start(){await this.clean();const configs=await this.getWebpackConfig();for(const config of configs){const defaultEntry=config.entry;config.entry=async(...args)=>{// @ts-ignore entry is always a functon
-const entrypoints=await defaultEntry(...args);const isClientCompilation=config.name==='client';await Promise.all(Object.keys(_onDemandEntryHandler.entries).map(async page=>{if(isClientCompilation&&page.match(_constants.API_ROUTE)){return;}const{serverBundlePath,clientBundlePath,absolutePagePath}=_onDemandEntryHandler.entries[page];const pageExists=await(0,_isWriteable.isWriteable)(absolutePagePath);if(!pageExists){// page was removed
+const entrypoints=await defaultEntry(...args);const isClientCompilation=config.name==='client';await Promise.all(Object.keys(_onDemandEntryHandler.entries).map(async page=>{if(isClientCompilation&&page.match(_constants.API_ROUTE)){return;}const{serverBundlePath,clientBundlePath,absolutePagePath}=_onDemandEntryHandler.entries[page];const pageExists=serverBundlePath.startsWith('pages/next/dist/pages')||(await(0,_isWriteable.isWriteable)(absolutePagePath));if(!pageExists){// page was removed
 delete _onDemandEntryHandler.entries[page];return;}_onDemandEntryHandler.entries[page].status=_onDemandEntryHandler.BUILDING;const pageLoaderOpts={page,absolutePagePath};entrypoints[isClientCompilation?clientBundlePath:serverBundlePath]=isClientCompilation?`next-client-pages-loader?${(0,_querystring.stringify)(pageLoaderOpts)}!`:absolutePagePath;}));return entrypoints;};}const multiCompiler=(0,_webpack.webpack)(configs);(0,_output.watchCompilers)(multiCompiler.compilers[0],multiCompiler.compilers[1]);// Watch for changes to client/server page files so we can tell when just
 // the server file changes and trigger a reload for GS(S)P pages
 const changedClientPages=new Set();const changedServerPages=new Set();const prevClientPageHashes=new Map();const prevServerPageHashes=new Map();const trackPageChanges=(pageHashMap,changedItems)=>stats=>{stats.entrypoints.forEach((entry,key)=>{if(key.startsWith('pages/')){entry.chunks.forEach(chunk=>{if(chunk.id===key){const prevHash=pageHashMap.get(key);if(prevHash&&prevHash!==chunk.hash){changedItems.add(key);}pageHashMap.set(key,chunk.hash);}});}});};multiCompiler.compilers[0].hooks.emit.tap('NextjsHotReloaderForClient',trackPageChanges(prevClientPageHashes,changedClientPages));multiCompiler.compilers[1].hooks.emit.tap('NextjsHotReloaderForServer',trackPageChanges(prevServerPageHashes,changedServerPages));// This plugin watches for changes to _document.js and notifies the client side that it should reload the page

package.json:

{
  "dependencies": {
    "next": "patch:next@10.1.2#./patches/next-patch.diff"
  }
}

Following up on what @ceefour said, Even on next@11.1.3-canary.76, I can only get things to work properly if both _error.tsx and 404.tsx exist. I feel like there’s some file existence logic still in play somewhere.

Hi, this has been updated in the latest version of Next.js v11.1.3-canary.76, please update and give it a try!

Yes the workaround still works in next 11, I’m just saying that it’s still not completely fixed in a way so you don’t have to use the workaround

@AndysonDK so what’s the current workaround?

The fix mentioned here seems to work fine: https://github.com/vercel/next.js/issues/21828#issuecomment-854318588 . Yep, you need to add the custom pages/404.tsx.

@meglio still doesn’t seem to be fixed in Next 11 sadly 😕

I second this. The patching approach by @strothj fixed the initial build issue and now next@10.1.3 works successfully with webpack@5.34.0 with yarn next dev. Hot reload isn’t working, which I imagine is the same issue as this.

is it planned to be fixed?