berry: [Bug] Package hoisting in workspaces are broken since v2.4.x

  • I’d be willing to implement a fix

Describe the bug

Before Yarn v2.4.0 all packages in the workspaces were properly hoisted to the root node_modules folder with node_linker.

However after updating to any version that is >= v2.4.0 hoisting is no longer working and every package in the workspace receives it’s own copies of all dependencies despite being compatible with others.

To Reproduce

Reproduction
// Sherlock reproduction
const fs = require('fs')

// Enable `node-modules` nodeLinker
fs.writeFileSync('.yarnrc.yml', 'nodeLinker: node-modules');

// Setup Workspaces
await packageJson({
  name: 'root', workspaces: ['packages/*'],
  dependencies: {
      "@webcomponents/webcomponentsjs": "~2.4.0",
      "rxjs": "~6.6.0",
      "zone.js": "~0.10.3"
  },
  devDependencies: {
      "@angular/core": "~9.1.12",
      "tslib": "~1.11.1",
      "typescript": "~3.8.3"
  },
  resolutions: {
      "typescript": "3.8.3"
  }
})
await packageJson({
  name: 'dashboard',
  dependencies: {
      "@angular/core": "^9.1.12",
      "@spryker/button": "^0.2.0-next.9",
      "rxjs": "^6.5.4",
      "tslib": "^1.10.0"
  }
}, { cwd: 'packages/dashboard' })
await packageJson({
  name: 'security',
  dependencies: {
      "@angular/core": "^9.1.12",
      "@spryker/card": "^0.2.0-next.2",
      "rxjs": "^6.5.4",
      "tslib": "^1.10.0"
  }
}, { cwd: 'packages/security' })
await packageJson({
  name: 'zed-ui',
  dependencies: {
      "@angular/core": "^9.1.12",
      "@spryker/button": "^0.2.0-next.9",
      "@spryker/web-components": "^0.2.1-next.5",
      "@webcomponents/webcomponentsjs": "^2.4.1",
      "rxjs": "^6.5.4",
      "tslib": "^1.10.0"
  }
}, { cwd: 'packages/zed-ui' })

// Perform install
await yarn('install')

// Check that all packages were hoisted in root
expect(fs.existsSync('node_modules/@angular/core')).toBe(true)
expect(fs.existsSync('node_modules/@spryker/button')).toBe(true)
expect(fs.existsSync('node_modules/@spryker/card')).toBe(true)
expect(fs.existsSync('node_modules/@spryker/web-components')).toBe(true)
expect(fs.existsSync('node_modules/@webcomponents/webcomponentsjs')).toBe(true)
expect(fs.existsSync('node_modules/rxjs')).toBe(true)
expect(fs.existsSync('node_modules/tslib')).toBe(true)
expect(fs.existsSync('node_modules/zone.js')).toBe(true)

// Check that there are no packages in workspaces packages
expect(fs.existsSync('packages/dashboard/node_modules')).toBe(false)
expect(fs.existsSync('packages/product-list/node_modulesr')).toBe(false)
expect(fs.existsSync('packages/product-offer/node_modules')).toBe(false)
expect(fs.existsSync('packages/profile/node_modules')).toBe(false)
expect(fs.existsSync('packages/sales-orders/node_modules')).toBe(false)
expect(fs.existsSync('packages/security/node_modules')).toBe(false)
expect(fs.existsSync('packages/zed-ui/node_modules')).toBe(false)

Also repro in separate repo:

  • Clone the following repository for repro: https://github.com/gund/yarn-worksapces-bug
  • Execute Yarn install: yarn install
  • Inspect root node_modules/@spryker folder - it only contains a few packages that are directly referenced in root package.json but it should contain a few dozen other packages from the nested workspace packages (located under packages/*/package.json)
Actual content of the root folder `node_modules/@spryker`:
➜  suite git:(demo-yarn-latest-bug) ls node_modules/@spryker
frontend-config.eslint    frontend-config.stylelint  jquery-query-builder  oryx-for-zed           sql-parser-mistic
frontend-config.prettier  frontend-config.tslint     nestable              sass-resources-loader
Expected content of the root folder `node_modules/@spryker`:
➜  suite git:(demo-yarn-latest-bug) ✗ ls node_modules/@spryker 
ajax-action                                   interception                       table.datasource.http
ajax-action.post-action.close-overlay         jquery-query-builder               table.datasource.inline
ajax-action.post-action.redirect              label                              table.feature.batch-actions
ajax-action.post-action.refresh-drawer        layout                             table.feature.editable
ajax-action.post-action.refresh-parent-table  locale                             table.feature.filters
ajax-action.post-action.refresh-table         logo                               table.feature.pagination
ajax-form                                     modal                              table.feature.row-actions
button                                        navigation                         table.feature.search
card                                          nestable                           table.feature.selectable
checkbox                                      notification                       table.feature.settings
chips                                         oryx-for-zed                       table.feature.sync-state
collapsible                                   pagination                         table.feature.title
data-serializer                               popover                            table.feature.total
date-picker                                   sass-resources-loader              table.filter.date-range
drawer                                        select                             table.filter.select
dropdown                                      sidebar                            table.filter.tree-select
form-item                                     sql-parser-mistic                  tabs
frontend-config.eslint                        styles                             textarea
frontend-config.prettier                      table                              toggle
frontend-config.stylelint                     table.action-handler.form-overlay  tree-select
frontend-config.tslint                        table.action-handler.html-overlay  unsaved-changes
grid                                          table.action-handler.url           unsaved-changes.guard.browser
header                                        table.column.chip                  unsaved-changes.guard.drawer
headline                                      table.column.date                  unsaved-changes.guard.navigation
html-renderer                                 table.column.image                 unsaved-changes.monitor.form
icon                                          table.column.input                 utils
input                                         table.column.select                web-components
input.password                                table.column.text

Screenshots

Missing dependencies install via Yarn v2.4.0:

image

All dependencies installed via Yarn v2.3.1:

image

Environment if relevant (please complete the following information):

  • OS: WSL Ubuntu on Windows 10 x64
  • Node version v14.15.0
  • Yarn version >=v2.4.0

Thanks!

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 5
  • Comments: 20 (10 by maintainers)

Most upvoted comments

Thank you for the feedback! I’ve created another issue so this one can be closed.

Let’s continue in #2727.

@gund It seems to me that another issue should be opened to discuss your concerns. The title of the issue is misleading. You are discussing the resolution algorithm and changes to the resolution algorithm. You are not proposing to change the hoisting algorithm in any way, the current hoisting algorithm will satisfy your needs and need not be changed.

The resolution algorithm is how package ranges are being resolved. The hoisting algorithm is how dependency graph with already resolved dependencies is being flattened on the file system.

I feel the discussion will be more constructive if it will be focused. Since no changes need to be done to hoisting algorithm. Its better focus on resolution algorithm only.

@gund Thanks, pulling out package.json into a separate repro would be helpful. Also, yes, Sherlock supports workspaces, for example see the Sherlock repro in the issue below: https://github.com/yarnpkg/berry/issues/1928