web: Problems with symlinked dependencies in monorepos and phantom dependencies

Since @open-wc/testing-karma is deprecated I’m migrating to @web/test-runner.

There are 2 big incompatibilities with monorepos.

esbuild duplicates symlinked lit-html instances

This problem is related to https://github.com/Polymer/lit-element/issues/390. Lit-html module should only be imported once, otherwise it will render [object Object]. The module loader should cache the first ‘lit-html’ call and server always the same module.

Note that preserveSymlinks: true is necessary. Without it web-test-runner will complain that @open-wc/testing is outside of the project directory.

web-test-runner referencees phantom dependencies

web-test-runner assumes that node_modules contains all explicit and implicit dependencies of the project. But this is incompatible with pnpm.

Reproduction

The problems can be reproduced in this repo:

https://github.com/javier-garcia-meteologica/test-runner-lit-hml-monopero-symlink-issue

$ git clone https://github.com/javier-garcia-meteologica/test-runner-lit-hml-monopero-symlink-issue
$ cd test-runner-lit-hml-monopero-symlink-issue
$ rush install
$ rush build
$ cd lit-element-ts-esbuild
$ yarn run test

I modified the code to log this.shadowRoot.innerHTML to the console. The tests will fail because lit-html is imported more than once and it’s rendering those pesky [object Object].

 🚧 Browser logs:
      <!---->[object Object]<!---->
      <!---->[object Object]<!---->
      <!---->[object Object]<!---->
      <!---->[object Object]<!---->

 ❌ MyElement > increases the counter on button click
      at: test/my-element.test.ts:16:44
      TypeError: Cannot read property 'click' of null
        at n.<anonymous> (test/my-element.test.ts:16:44)

rush uses pnpm as a package manager by default, but this repository had to switch to yarn so that web-test-runner finds phantom dependencies.

To replicate the problem of phantom dependencies, edit the file <monorepo_root>/rush.json, uncomment "pnpmVersion" and comment out "yarnVersion".

"pnpmVersion":"4.14.4",
//"npmVersion":"4.5.0",
//"yarnVersion":"1.9.4",

Now purge the monorepo and rebuild

$ rm <monorepo_root>/common/config/rush/yarn.lock
$ rush update --full --purge
$ rush build

Running tests will fail and the errors will look like these:

TypeError: Failed to resolve module specifier "@open-wc/testing-helpers". Relative references must start with either "/", "./", or "../".
TypeError: Failed to resolve module specifier "chai/chai.js". Relative references must start with either "/", "./", or "../".

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

Finally I found out which were the remaining problems and how to solve them. First, preserveSymlinks: true should NOT be used because the only way to solve non-hoisted pnpm dependencies is to resolve the symlinks and, from the resolved directory, traverse parent directories looking up for node_modules with those dependencies. Thanks to @LarsDenBakker package resolution in pnpm works fine without preserveSymlinks.

Therefore, the configuration should be:

export default {
  nodeResolve: { dedupe: ['lit-html'] },
  plugins: [esbuildPlugin({ ts: true })]
}

This will almost work, except it won’t resolve lit-html.

Error while transforming ../common/temp/node_modules/.pnpm/registry.npmjs.org/lit-element/2.4.0/node_modules/lit-element/lit-element.js: Could not resolve import "lit-html/lib/shady-render.js".
  54 |  * @packageDocumentation
  55 |  */
> 56 | import { render } from 'lit-html/lib/shady-render.js';
     |                        ^
  57 | import { UpdatingElement } from './lib/updating-element.js';
  58 | export * from './lib/updating-element.js';
  59 | export * from './lib/decorators.js';

That’s because @rollup/node-resolve assumes that deduplicated dependencies are placed in the main node_modules directory of our project.

https://github.com/rollup/plugins/blob/0e4f0142a9c44074ae8a5df86f9442b119a9978c/packages/node-resolve/src/index.js#L109

      const baseDir = !importer || dedupe(importee) ? rootDir : dirname(importer);

Perhaps lit-html@2 won’t need to be deduped, but so far we have to deal with it. So if we have deduplicated dependencies we need to add them as dependencies or devDependencies of our main project, even if they are not used directly.

# If you use rush
rush add -p lit-html --dev
# If you use pnpm directly to manage your monorepo
pnpm add -D lit-html

And that’s all, now everything should work!

javier-garcia-meteologica on Jan 13, 2021

Can you try with the latest test-runner/dev-server? I’ve made it possible to resolve modules to any location.

LarsDenBakker on Dec 20, 2020