next.js: Turbopack can't locate serverExternalPackages if installed via pnpm and are child dependencies of other installed modules

Link to the code that reproduces this issue

https://github.com/jmikrut/nextjs-68805

To Reproduce

  1. Install dependencies via pnpm
  2. Run pnpm dev --turbo
  3. Go to http://localhost:3000/admin

Current vs. Expected behavior

Turbopack fails to compile, and the terminal blows up with errors such as the following:

Package @libsql/client (serverExternalPackages or default list) can't be external
The request @libsql/client matches serverExternalPackages (or the default list), but it can't be external:
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install the package into the project directory.

If you install dependencies with Yarn, or with npm i --legacy-peer-deps, the dependencies are located correctly with Turbopack.

Note that all dependencies which error are indeed installed as dependencies of installed packages. Webpack works, but Turbopack does not.

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 24.0.0: Wed Jul 31 21:51:10 PDT 2024; root:xnu-11215.0.199.501.2~1/RELEASE_ARM64_T6000
  Available memory (MB): 32768
  Available CPU cores: 10
Binaries:
  Node: 20.11.1
  npm: 10.2.4
  Yarn: 1.16.0
  pnpm: 8.15.7
Relevant Packages:
  next: 15.0.0-canary.111 // Latest available version is detected (15.0.0-canary.111).
  eslint-config-next: 15.0.0-canary.104
  react: 19.0.0-rc-06d0b89e-20240801
  react-dom: 19.0.0-rc-06d0b89e-20240801
  typescript: 5.5.4
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Turbopack

Which stage(s) are affected? (Select all that apply)

next dev (local)

Additional context

This appears to have broken in next@15.0.0-canary.84. Worked before that.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 66
  • Comments: 53 (34 by maintainers)

Commits related to this issue

Most upvoted comments

Encountered the same issue on next@15.0.0-rc.1 with import-in-the-middle & require-in-the-middle, which seem to be dependencies of @opentelemetry/instrumentation, which in turn is a dependency of @sentry/nextjs. image

I don’t understand what is there for users to fix. Example 1:

 ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.54.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node
Package require-in-the-middle can't be external
The request require-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install require-in-the-middle from the project directory.

Is this suggesting that I install this require-in-the-middle package directly into the main project? I never had to install dependencies of dependencies like this before with any bundler.

Example 2:

 ⚠ ./packages/auth/dist/lib
Package oslo can't be external
The request oslo matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install oslo from the project directory.

As you can guess from the path, this dependency comes from the build folder of another package in a monorepo. But the main project only imports the auth package "@project/auth" : "workspace:^". The oslo depedency is already installed in the auth project.

Finally, here is an example of my terminal log, just to give you a feel of how annoying this is in terms of DX:

 ⚠ ./packages/storage/dist/provider/s3
Package @aws-sdk/client-s3 can't be external
The request @aws-sdk/client-s3 matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install @aws-sdk/client-s3 from the project directory.


 ⚠ ./node_modules/.pnpm/@prisma+client@5.22.0_prisma@5.22.0/node_modules/.prisma/client
Package @prisma/client can't be external
The request @prisma/client/runtime/library.js matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install @prisma/client from the project directory.


 ⚠ ./packages/db/src
Package @prisma/client can't be external
The request @prisma/client matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install @prisma/client from the project directory.


 ⚠ ./packages/db/src/zod
Package @prisma/client can't be external
The request @prisma/client matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install @prisma/client from the project directory.


 ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.52.1_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node
Package import-in-the-middle can't be external
The request import-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install import-in-the-middle from the project directory.


 ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.53.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node
Package import-in-the-middle can't be external
The request import-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install import-in-the-middle from the project directory.


 ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.54.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node
Package import-in-the-middle can't be external
The request import-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install import-in-the-middle from the project directory.


 ⚠ ./node_modules/.pnpm/@sentry+node@8.37.1/node_modules/@sentry/node/build/cjs/sdk
Package import-in-the-middle can't be external
The request import-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install import-in-the-middle from the project directory.


 ⚠ ./packages/auth/dist/lib
Package oslo can't be external
The request oslo matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install oslo from the project directory.


 ⚠ ./packages/auth/dist/lib
Package oslo can't be external
The request oslo/crypto matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install oslo from the project directory.


 ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.52.1_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node
Package require-in-the-middle can't be external
The request require-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install require-in-the-middle from the project directory.


 ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.53.0_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node
Package require-in-the-middle can't be external
The request require-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install require-in-the-middle from the project directory.


 ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.54.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node
Package require-in-the-middle can't be external
The request require-in-the-middle matches serverExternalPackages (or the default list).
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install it into the project directory by running npm install require-in-the-middle from the project directory.

Good news, it was already reported internally—we are taking a look!

Hey Next.js team - thank you for making these warnings instead of full-on errors. That is a good step.

Everything works and is fully functional at this point but WOAH there are lots of warnings being logged. Looks like the warnings log on every request.

Maybe you could consider adding a flag to silence these specific warnings in the Next.js config? That would be huge and with that I’d be all set on this issue.

@simoami Try this in .npmrc:

public-hoist-pattern[]=*import-in-the-middle*
public-hoist-pattern[]=*require-in-the-middle*

So is the solution (for now) to move off pnpm and back to yarn or npm?

Hey @samcx

I created a very, very minimal reproduction of this issue here: https://github.com/AlessioGr/very-minimal-turbo-bug-repro

It’s a blank, fresh Next.js project that includes a small, local package in the mydep folder. You can (don’t have to, it’s pre-built in the repo) build & pack mydep by running cd mydep && pnpm install && pnpm build - then go back to the root of the project and run rm -rf pnpm-lock.yaml && pnpm i.

The mydep package is very simple:

import { drizzle } from 'drizzle-orm/libsql';
import { createClient } from '@libsql/client';

export async function test(dbURL) {
    const client = createClient({ url: dbURL });
    const db = drizzle(client);
    console.log('db', db);
}

It has the following dependencies:

    "@libsql/client": "0.6.2",
    "drizzle-orm": "0.32.1",

Calling test connects to an SQLite db and then logs the db object to console.

Within the Next.js app/page.tsx we’re using that package like this:

import { test } from 'mydep';

export default function Home() {
  test('file:./test-sqlite.db')

  return (
    <p>
      test
    </p>
  );
}

Super simple! External package uses drizzle-orm and @libsql/client to connect to sqlite - that external package is then used within the page.tsx.

When running pnpm dev & opening the page, it works fine. db object is logged to console, and a test-sqlite.db is created.

When running pnpm dev --turbo you will encounter the serverExternalPackages error.

I shouldn’t have to install @libsql/client in the consuming project for turbo to work. @libsql/client is a dependency (not peerDependency) of mydep and scoped by pnpm to mydep. It can thus be used by mydep - but we cannot import @libsql/client directly by the consuming project. And this is good! That’s how it should be. It’s implementation detail of mydep, and in our consuming project, we can only use what mydep exports.

But Turbopack currently does not respect this. It would be a nightmare for package authors to force their users to install their packages’ own dependencies

@simoami Try this in .npmrc:

public-hoist-pattern[]=*import-in-the-middle*
public-hoist-pattern[]=*require-in-the-middle*

this worked for me

A temporary workaround is to use public-hoist-pattern[] in .npmrc.

For example, for me I was getting the error for the following packages; keyv and rimraf. So I updated my .npmrc file in my workspace root to include:

public-hoist-pattern[]=*keyv*
public-hoist-pattern[]=*rimraf*

Then just run pnpm install again.

Prisma

Have the same issue as this, I had to de-activate turbopack mode for my project for now

@mischnic your notes make sense, and we understand your reasoning, but we flatly do not think that the current Turbopack functionality is acceptable. (AlessioGr is on my team).

We would rather the onus of mismatching versions be on the developer, so they can be responsible for the risk you’ve described rather than Turbopack becoming non-functional even in cases where versions match.

It is strange that there are inconsistencies with package managers (dev needs to now think about which package manager they use in order to use Turbopack).

How does your implementation of Webpack currently accomplish this? This all works in Webpack correctly.

The Node ecosystem would require a dependent package based on the context that it was required in. Maybe your current Webpack implementation patches in / uses a workaround to do something similar?

Example, if we install DependencyA which installs and relies on DependencyB, then DependencyA would require DependencyB and it would be sourced from DependencyA’s node_modules. Right?

As Turbopack works now, a require is always a top-level require. I think this is where the disconnect is. I also understand the fact that Next.js bundles and the post bundle will indeed technically be a top-level require, but that is an implementation detail that I think Turbopack should solve for.

I really think that if we have to tell people to install all sub-dependencies of our packages in their repo, they will simply be inclined to continue using Webpack rather than Turbopack.

In any case, thank you for your continued insight! We just want to voice our concern and ensure that we know how to proceed here (if we do indeed have to tell end-users of Payload to install our sub-dependencies manually or not).

+1 Experiencing the same error with Prisma by following this guide

./node_modules/.pnpm/@prisma+client@5.18.0_prisma@5.18.0/node_modules/@prisma/client/default.js

Package @prisma/client (serverExternalPackages or default list) can't be external
The request @prisma/client matches serverExternalPackages (or the default list), but it can't be external:
The request could not be resolved by Node.js from the project directory.
Packages that should be external need to be installed in the project directory, so they can be resolved from the output files.
Try to install the package into the project directory.

Same issue here. Using Payload beta 116.

▲ Next.js 15.0.0-rc.1 (turbo)

  • Local: http://localhost:3000
  • Environments: .env.local, .env
  • Experiments (use with caution): · turbo · reactCompiler

✓ Starting… ✓ Ready in 1138ms ○ Compiling / … ✓ Compiled / in 9.3s ⨯ ./node_modules/.pnpm/pg@8.11.3/node_modules/pg/lib/index.js Package pg (serverExternalPackages or default list) can’t be external The request pg matches serverExternalPackages (or the default list), but it can’t be external: The request could not be resolved by Node.js from the project directory. Packages that should be external need to be installed in the project directory, so they can be resolved from the output files. Try to install the package into the project directory.

./node_modules/.pnpm/pino@9.3.1/node_modules/pino/pino.js Package pino (serverExternalPackages or default list) can’t be external The request pino matches serverExternalPackages (or the default list), but it can’t be external: The request could not be resolved by Node.js from the project directory. Packages that should be external need to be installed in the project directory, so they can be resolved from the output files. Try to install the package into the project directory.

./node_modules/.pnpm/pino-pretty@11.2.1/node_modules/pino-pretty/index.js Package pino-pretty (serverExternalPackages or default list) can’t be external The request pino-pretty matches serverExternalPackages (or the default list), but it can’t be external: The request could not be resolved by Node.js from the project directory. Packages that should be external need to be installed in the project directory, so they can be resolved from the output files. Try to install the package into the project directory. ○ Compiling /_error … ✓ Compiled /_error in 3.5s GET / 500 in 13117ms

Hello,

Any updates for this issue ?

I’ve already this error…

Thanks you

Capture d’écran 2024-11-06 à 17 16 36

I’ve found this issue too and initially thought that it was a hoisting issue. But it’s apparently caused by mismatching version of the dependency that’s being used by different workspace. A silly mistake

So for example, I had a monorepo with this structure:

  • apps/web
    • will contain a node_module with firebase-admin@10
    • package.json (uses firebase-admin@10)
  • packages/core
    • no node_module
    • package.json (uses firebase-admin@11)
  • node_module contain firebase-admin@11
  • package.json (set the workspace)

The symptom of mismatching version would have individual workspace with seemingly duplicate package in node_modules

The solution is to change the firebase-admin dependency in apps/web to use the same version with packages/core

While this may not be the replicated the original problem of the thread, hopefully this helps for those who are stuck.

@jmikrut The solution on the PNPM side for Sqlite adapter and turbopack is to publicly hoist *@libsql*. Perhaps this should be ran as a default option within CPA if init target is pnpm ?

legacy-peer-deps=true
public-hoist-pattern[]=*@libsql*

@jmikrut Can you share a screenshot of these warnings?

I would usually lean on fixing the warning instead of outright silencing it. Silencing a warning may lead to the warning never getting fixed.

I was getting the following warning when running turbo with sentry installed ⚠ ./node_modules/.pnpm/@opentelemetry+instrumentation@0.54.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/instrumentation/build/esm/platform/node Package import-in-the-middle can't be external The request import-in-the-middle matches serverExternalPackages (or the default list). The request could not be resolved by Node.js from the project directory. Packages that should be external need to be installed in the project directory, so they can be resolved from the output files. Try to install it into the project directory by running npm install import-in-the-middle from the project directory.

Then I installed import-in-the-middle and added the following to my next config transpilePackages: ["import-in-the-middle"],

This at least hid the warnings for me

I have moved to Bun because of this issue, and there have been no issues so far. It’s also affecting other libraries, not just Drizzle.

A general update here: we made this a warning (also in the just released Next.js 15) instead of a hard error.

@GustavoOS Could you try Next.js 15 and if you still get a could not resolve error, open a new issue with a reproduction?

@samcx I updated with a proper repro that simplifies and does not require environment variables. Should just need to clone the newly linked repro above and run it with turbo to see it fail.

Hey @samcx you need to load up the admin UI (the spot that uses the dependencies in question). I forgot to add that to the reproduction instructions - good catch.

Updated! If you’re still not able to repro, I also noticed that you’re on pnpm 9.7.0 and I’m on 8.15.7. Might have to do with it.