webpack: Duplicate modules - NOT solvable by `npm dedupe`

It doesn’t look like it was ever provided, so I created a repo that shows a simple example of this issue: https://github.com/franjohn21/dedupe-example

FWIW the code being webpacked in that example does not use harmony modules, so ideally webpack 2/3 could maintain parity with webpack 1.

We consistently see higher (>10%) bundle sizes in production apps at NerdWallet when migrating from webpack 1 -> webpack 3 mostly due to this issue.

Thing is, I didn’t choose to install the package twice, and there is no way for me to “fix” this at the package manager level, as you’ve suggested in other issue threads.

Wepback used to solve this problem in v1 with DedupePlugin, so this seems like a regression of sorts. The docs say that the DedupePlugin is no longer needed, which implies that it will do some kind of deduping of its own.

Was any official resolution to this issue ever suggested?

I’d also love some sort of solution for this - I’ve got several packages with the same dependency and it’s being loaded twice.

yarn install --flat has removed the duplicates from the subdirectories but they’re still duplicated in the bundle.

I seem to have this issue when using webpack in a mono-repo managed by lerna. Various packages within the mono-repo depend on the same version of the module yaml and it appears to be being included in the bundle more than once. Any advice would be welcomed.

The least disruptive solution for me has been to take advantage of Webpack’s resolve.alias option. I only have under a handful of large duplicated dependencies, so manual intervention works, but certainly isn’t scalable.

I understand that Webpack’s DedupePlugin has been deprecated/removed for awhile, but I don’t understand what would have been too hacky about it. If I have two imports with the same name, and both package.json are the same version, and the file size and contents are identical… it seems like Webpack could safely deduplicate this without consequence.

What makes the problem even worse, though, is the difficulty in its identification. duplicate-package-checker-webpack-plugin won’t detect multiple copies of the same version of a package, as discussed in https://github.com/darrenscerri/duplicate-package-checker-webpack-plugin/pull/21.

So to really know this is occurring, you need to periodically inspect your stats.json with a tool like https://webpack.github.io/analyse/, sort by file size, and manually inspect. Since npm dedupe won’t work in all cases, it seems reasonable to shift the responsibility to webpack.

I will put together a repo that demonstrates this issue in its simplest form. Give me a day to do this.

Furthermore, this has the potential to cause sudden bumps in the package size, without much warning.

Let’s say that I have an app using a popular lib, like lodash. Lots of dependencies (let’s say 20) use lodash and they’re all using compatible versions, let’s say everyone is reuqesting ~1.1.0. My app requires ~1.1.0, and everything is great - I have 1 copy of v1.1.X in my bundle.

Next day, a version 1.2 of lodash is released, and I update my app to use it. Now, when my app builds, I end up with 1 copy of v1.2 and 20 copies of v1.1.X.

Whilst that may not be a bug, and sure, webpack might be behaving like specced, it’s not very helpful from a tool that is supposed to help with building optimised versions of application bundles for the web.

I had to use resolve.alias to get around this issue.

    resolve: {
      extensions: [ '.js', '.jsx' ],
      alias: {
        'react': path.resolve(__dirname, '../../node_modules/react/')
      }
    },

I was able to achieve the desired effect described in this issue with some creative use of the NormalModuleReplacementPlugin. Note that I tested this solution with webpack 3.

My project depends on immutable@3.8.1, draft-js and draft-js-plugins-editor. draft-js and draft-js-plugins-editor both depend on immutable@~3.7.4. Here’s how I deduped immutable so that only two versions are included in my output:

new NormalModuleReplacementPlugin(/^immutable$/, resource => {
	if (resource.context.includes('node_modules/draft-js-plugins-editor')) {
		resource.request = 'draft-js/node_modules/immutable';
	}
});

I’m basically instructing webpack that any call to import ... from 'immutable' inside draft-js-plugins-editor should be replaced with import ... from 'draft-js/node_modules/immutable'.

Wouldn’t solution the to this issue be rather trivial in nature? In the directory structure @tomhicks-bsf provided, the same library version is installed twice, and while you may disagree whether this should be done, there should be nothing complex about handling this scenario and bundling in the duplicated library only once.

All it would take for this to work is for webpack to check for modules’ contents to verify whether two (or more) imported modules are in fact the same.

My blind guess is that path at which the module is being resolved in currently taken into account, and this is why the same version of a library would be bundled in twice instead of once.

Looks like it indeed used to work as indicated by the docs at https://github.com/webpack/docs/wiki/optimization#deduplication

@terencechow in your case when you have the same version of dependency this can be fixed by removing node_modules directory and installing modules again with npm or yarn command. Both commands will place immutable to /node_modules and this package will not be duplicated in sub-directories.

For other cases when we have 3 copies of dependency v1 and 3 copies of v2 I wrote a small plugin https://github.com/RoboBurned/dedup-resolve-webpack-plugin Note: It is completely untested and draft. Was used only in one project. It can brake all.

duplicate of https://github.com/webpack/webpack/issues/985

Having the same problem, got a mono repo with multiple packages, all packages depend on one core package which contains most of the dependencies. However after building every entry point as well as the vendor chunk contains the chunk that was supposed to be split, rendering splitChunks useless.

I would say this is a huge bug since the only real reason to split chunks is caching… I want to split a dependency from the rest of my bundle so it doesn’t have to be redownloaded every time my bundle changes. However instead of reducing this problem webpack is now actually increasing it…

Result should be: vendor.js dep.js app1.js app2.js app3.js

Reality is: vendor.js dep.js dep.js app1.js dep.js app2.js dep.js app3.js dep.js

Then there are 2 projects, project 1 contains app1 and app2, project 2 contains app3. Now when I’m download project 1 in the browser I am actually including (and therefore downloading) dep.js FOUR times instead of one. Twice in vendor.js, once in app1.js, once in app2.js. When I download project 2 it includes dep.js THREE times, twice in vendor.js, once in app3.js.

Seems to me that this is a bug; in no way or form should webpack ever increase bundle size compared to the source (aside from boilerplating and such which is negligible). With this kind of behavior it actually makes no sense to use splitChunks because it’s actually increasing my bundle and chunk sizes. I am inclined to remove it entirely and find some other solution. The only possibility I can still imagine is by splitting every single package from node_modules to a seperate file with the name and version of the package…

FYI: In my case these are the exact same module (only one version exists in the filesystem), only difference is that they are shared by multiple packages and therefore symlinked to those packages.

@maciej-gurban that’s reasonable in some cases, but in other cases yarn workspaces doesn’t resolve this issue. Say you have nine dependencies, three each depending on version 1.x, 2.x, and 3.x of some-lib. There’s nothing yarn can do to not at a very minimum pull in three copies of one of those versions of some-lib.

@benthemonkey tried out your solution and it worked for me, though I’m a little scared what will happen when paths/versions of things change or someone else has to discover what’s going on haha

We’ve run in to this issue also, where we need to have two major versions of a library included in our bundle but don’t want to have multiple copies beyond those two. In our case, our app was using version 1.x of a large library (let’s call it lib) and also depended on many packages, let’s say a, b, and c, which also depended on 1.x of lib. When we bumped our app to 2.x, all of the sudden each of a, b, and c all pull in their own copy lib@1.x which made out bundle size increase by multiple megabytes.

@benthemonkey’s solution looks clever, I’ll have to give it a shot, but in the end it’s a somewhat brittle hack that introduces complexity to the build. Some sort of officially supported solution to this issue (or a well-supported plugin) could be very beneficial.

Thanks for sharing your plugin work @RoboBurned, curious if other people have had any luck with it so far?