sst: pnpm not supported

pnpm finally supported in v2!

We’ll be releasing a public beta of the new CLI soon. With pnpm support.

Yeah so it’s an AWS CDK issue but we are working on a new CLI that’ll fix this.

I don’t see how can that be a CDK issue where, in fact, I’ve worked on many monorepo projects that use AWS CDK + PNPM just fine, but attempting to configure a brand new basic monorepo with SST + PNPM leads to all sorts of errors like:

Preparing your SST app
node:internal/process/esm_loader:94
    internalBinding('errors').triggerUncaughtException(
                              ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@serverless-stack/core' imported from /Users/me/projects/acme-project/packages/backend/.build/run.mjs
Did you mean to import @serverless-stack+core@1.18.4/node_modules/@serverless-stack/core/dist/index.js?
    at new NodeError (node:internal/errors:371:5)
    at packageResolve (node:internal/modules/esm/resolve:930:9)
    at moduleResolve (node:internal/modules/esm/resolve:976:18)
    at defaultResolve (node:internal/modules/esm/resolve:1078:11)
    at ESMLoader.resolve (node:internal/modules/esm/loader:530:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:251:18)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:79:40)
    at link (node:internal/modules/esm/module_job:78:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}


There was an error synthesizing your app.

On the surface, it looks like SST is just relying too much on node_modules being a “local” folder, rather than being hoisted to the top of the monorepo. Because if we install the dependencies locally, SST magically starts working.

Not trying to point a finger at SST, just want to make sure the issue is addressed at the right place rather than waiting for a magical upstream bugfix to come down from AWS CDK, that will never happen.

EDIT: To better illustrate what I mean.

pnpm install
cd packages/sst && sst deploy

Leads to synthesis errors.

Whereas:

pnpm install --shamefully-hoist
cd packages/sst && sst deploy

Works fine.

In a different project that uses AWS CDK as the framework for “Infrastructure as code”, both approaches work fine:

pnpm install # or pnpm install --shamefull-hoist
cd packages/aws-cdk && npx cdk deploy --all

I can prepare a minimally reproducable repo of course, but I think this is enough information as a starting point to further analysis.

I need to do another deep dive into pnpm now that CDK v2 is out, some of what I say below may be outdated.

Next-gen bundlers like yarn2 + pnpm enforce certain constraints that some projects took advantage of. For example, in CDK if they’re including an asset that gets sent up with your cloudformation, they would resolve it using __dirname. In yarn2 the filesystem for dependencies is “virtual” and you need to be using module resolution to find things. This leads to some things not working correctly.

If it works for you then I’d probably continue to use it until you hit something that isn’t compatible. I’ll do another pass to see if I can test more thoroughly with cdkv2

@thdxr would be great to have a little more context on this - we have been successfully using CDK + pnpm + rush for the last year or so. We have a demo projects also deployed successfully with sst but we aren’t using all of sst’s features (e.g. add-cdk, we have typeCheck and lint disabled since we do those separately). Do you know what exactly isn’t supported? Or is it just that there are no guarantees?

Hey @riccardogiorato unfortunately we cannot support PNPM at this time due to CDK not supporting PNPM. We decided to wait for them to prioritize fixing their incompatibilities before we fix ours

Moving to pnpm with a Rush.js monorepo setup broke my SST app.

I have a lot of microservices and want to use pnpm to ensure consistency between services with module versions and to reduce downloads/build time

I think the solution here is not not assume the location of tsc in relation to the node_modules but to do a fs.stat and check.

https://github.com/serverless-stack/serverless-stack/blob/ce288401dde6fb237e504513b259cefba6fdb639/packages/cli/scripts/util/cdkHelpers.js#L69

const subpaths = [
  path.join(".bin", "tsc"),
  path.join("typescript", "bin", "tsc")
];

const testTscPath = async (subpath: string): Promise<string> => {
  const testPath = path.join(matches[1], subpath);

  await fs.promises.stat(testPath)

  return testPath;
}

try {
  // if not targetting ES2021, use the `promise.any` shim on npm
  return await Promise.any(subpaths.map(testTscPath));
}
catch(e) {
  console.log("No tsc found");
}