webpack: Webpack: How to don't bundle node_modules, but use them normally in node.js?

Architecture

I would like to share code between client and server side. I have defined aliases in the webpack config:

resolve: {
    // Absolute paths: https://github.com/webpack/webpack/issues/109
    alias: {
        server : absPath('/src/server/'),
        app    : absPath('/src/app/'),
        client : absPath('/src/client/'),
    }
},

Problem

Now on the server side I need to include webpack in order to recognize the correct paths when I require a file. For example

require('app/somefile.js')

will fail in pure node.js because can’t find the app folder.

What I need (read the What I need updated section)

I need to be able to use the webpack aliases. I was thinking about making a bundle of all the server part without any file from node_modules. In this way when the server starts it will use node_modules from the node_modules folder instead of a minified js file (Why? 1st: it doesn’t work. 2nd: is bad, because node_modules are compiled based on platform. So I don’t want my win files to go on a unix server).

Output:

  • Compiled server.js file without any node_modules included.
  • Let the server.js to use node_modules;

What I need updated

As I’ve noticed in https://github.com/webpack/webpack/issues/135 making a bundled server.js will mess up with all the io operation file paths.

A better idea would be to leave node.js server files as they are, but replace the require method provided with a custom webpack require which takes in account configurations such as aliases (others?)… Can be done how require.js has done to run on node.js server.

What I’ve tried

By adding this plugin in webpack

 new webpack.optimize.CommonsChunkPlugin(/* chunkName= */"ignore", /* filename= */"server.bundle.js")

Entries:

entry: {
    client: "./src/client/index.js",
    server: "./src/server/index.js",
    ignore: ['the_only_node_module'] // But I need to do that for every node_module
},

It will create a file server.js which only contains my server code. Then creates a server.bundle.js which is not used. But the problem is that webpack includes the webpackJsonp function in the server.bundle.js file. Therefore both the client and server will not work.

It should be a way to just disable node_modules on one entry.

What I’ve tried # 2

I’ve managed to exclude the path, but requires doesn’t work because are already minified. So the source looks like require(3) instead of require('my-module'). Each require string has been converted to an integer so it doesn’t work.

In order to work I also need to patch the require function that webpack exports to add the node.js native require function (this is easy manually, but should be done automatically).

What I’ve tried # 3

In the webpack configuration:

{target: "node"}

This only adds an exports variable (not sure about what else it does because I’ve diffed the output).

What I’ve tried # 4 (almost there)

Using

require.ensure('my_module')

and then replacing all occurrences of r(2).ensure with require. I don’t know if the r(2) part is always the same and because of this might not be automated.

Related

Thanks

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Reactions: 46
  • Comments: 25 (3 by maintainers)

Most upvoted comments

You can also use webpack-node-externals:

var nodeExternals = require('webpack-node-externals');
...
module.exports = {
    ...
    target: 'node', // in order to ignore built-in modules like path, fs, etc. 
    externals: [nodeExternals()], // in order to ignore all modules in node_modules folder 
    ...
};

Holy crap! Luckily, somehow found a fix here:

kriasoft/react-starter-kit#249

externals: fs.readdirSync("node_modules"),

@acarl005 I personally use a lot of aliases in Webpack that point to different parts of my app:

// webpack.config.js
...
resolve: {
  alias: {
    'components': './src/components',
    'services': './src/services',
    ...
  }

// Root.js
import "TextBox" from "components/TextBox";

If I’m externalizing using a regex, it will leave un-bundled those modules as well, since they are falsely recognized as “not starting with period”. (It is also explained here).

Going through the actual node modules guarantees the most stable externals config.

@bitplanets You initial approach would work too, if you use the externals option to exclude node_modules. i. e.

externals: fs.readdirSync("node_modules").map(function(module) {
  return "commonjs " + module
})

Why blacklist a bunch of npm packages in externals? The cleanest way is to just use a regexp that matches things that don’t start with a period. All of your own modules should be a relative path and should start with either ./ or ../.

module.exports = {
  entry: { ... },
  target: 'node',
  output: { ... },
  externals: /^[^.]/,
  module: {
    loaders: [
      {
        test: /\.js$/,
        loader: 'babel',
        query: {
          presets: [ 'es2015', 'stage-0' ],
        },
      },
    ],
  },
  context: __dirname,
  node: {
    __filename: true,
  }
}

EDIT: Thanks to some helpful comments, I now realize this simple strategy is not viable when using Webpack’s aliases.

Another version here:

      externals: function(context, request, callback) {
        // Absolute & Relative paths are not externals
        if (request.match(/^(\.{0,2})\//)) {
          return callback();
        }

        try {
          // Attempt to resolve the module via Node
          require.resolve(request);
          callback(null, request);
        } catch(e) {
          // Node couldn't find it, so it must be user-aliased
          callback();
        }
      },

Just don’t tell webpack about the node_modules you want to use: eval(“require”)(“module with dynamic require statements or optional dependendencies…”)

In my opinion, we don’t need bundle in node environment. If the result are many files, we don’t need tree-shaking as well.

Obviously, it would be useful to have this context in a function of externals handling like in loaders. The context would have resolve or resolveSync at least.

externals: [
    function(context, request, callback) {
        // get absolute path to required file
        const resourcePath = this.resolveSync(context, request);

        // don't bundle node_modules
        if (/node_modules/.test(resourcePath)) {
            return callback(null, 'commonjs ' + request);
        }
        callback();
    }
]

Is there any problem to add the feature?

@MikeBailleul Other advantages: You have a single file to distribute that will just run. No npm install needed.

And a single file app starts faster (significantly so in some cases) because require is slow.

Also, going through the ‘compile’ process for the server side can help you use ES2015 features and still support older Node JS versions (same as with browsers basically).

You can minify it and have smaller footprint etc etc.

It all depends on what you want to achieve. But there can be many such advantages depending on what your requirements are.

the solution provided by @liady seems pretty simple to me.

Yes it looks like a very nice module! But still it is a complete third-party library just to set one of the many Webpack config settings… It kinda shows that Webpack config is a bit too complex sometimes when you need an external module for such things.

But using a bundle on the server side has some advantages. Doing HMR on the server side during development is one. It would be great if there was a simpler way to just tell webpack 'exclude everything in `node_modules``. Currently it’s a difficult configuration item to get right.

Just wasted an hour on this. I thought that resolve.root and resolve.alias worked on the server but it doesn’t.

So much for sharing code between both server & client 😦

Unless there’s a fix for this, it’d probably be best to put a note in the docs about how it’s target: "web" only.

There’s a edge case not handled by the solution proposed by @sokra because modules with “/” in their name will not get properly replaced. You need to use a function that identifies slashes and checks if the base name is in the node_modules folder also:

var Fs = require('fs')
var nodeModules = {}
Fs.readdirSync('node_modules').forEach(function (module) {
  if (module !== '.bin') {
    nodeModules[module] = true
  }
})
var nodeModulesTransform = function(context, request, callback) {
  // search for a '/' indicating a nested module
  var slashIndex = request.indexOf("/");
  var rootModuleName;
  if (slashIndex == -1) {
    rootModuleName = request;
  } else {
    rootModuleName = request.substr(0, slashIndex);
  }

  // Match for root modules that are in our node_modules
  if (nodeModules.hasOwnProperty(rootModuleName)) {
    callback(null, "commonjs " + request);
  } else {
    callback();
  }
}

...
externals: nodeModulesTransform

You can see a complete webpack config with this working here: https://gist.github.com/lingz/14ad8ea44998f5009d5d

Thanks. But I think that not bundling is better because I don’t have to care about relative paths. (Almost 90% sure that I will migrate my 10K lines project to webpack+gulp! I’m coming from require.js+grunt)

I faced trouble like this (if I understood correctly)… I resolved it by using enchanced-require module.

This is by bootstrap.js I’m using express. This file is launches then bin/www and exposes custom require func to him and whole app.

No precompiled version needed for serverside…

And this is bootstrap.js file. Check this out:

/*
 *  APPLICATION BOOTSTRAP
 */

'use strict';
var webpack = require('webpack');
var path = require('path');
var _ = require('lodash');

global.IS_CLIENT = false;
global.SIDE = 'backend';

var options = {
  debug: true,
  recursive: true,
  resolve: {
    root:  path.resolve(__dirname, '..'),
    extensions: ['', '.js', '.jsx'],
    modulesDirectories: ["node_modules", "bower_components"],
    alias: {
      'models': path.join(__dirname, '..' ,'models'),
      'lib': path.join(__dirname, '..' ,'lib'),
      'config': path.join(__dirname, '..' ,'lib/config'),
      'flux': path.join(__dirname, '..','src/scripts/flux'),
      'routes': path.join(__dirname, '..','src/scripts/routes'),
      'components': path.join(__dirname, '..','src/scripts/components'),
      'scripts': path.join(__dirname, '..','src/scripts'),
      'styles': path.join(__dirname, '..','src/styles')
    }
  },
  module: {
    loaders: [{
      test: /react-tabs.*\.js$/,
      loader: 'jsx-loader'
    },{
      test: /\.jsx$/,
      loader: 'jsx-loader?harmony'
    }, {
      test: /\.css$/,
      loader: 'css-loader'
    }, {
      test: /\.(png|jpg|gif)$/,
      loader: 'raw-loader'
    }, {
      test: /icon-.+\.(svg)$/,
      loader: 'raw-loader'
    }, {
      test: /\.styl$/,
      loader: 'css-loader!stylus-loader'
    }, { test: /\.woff(\?v=[0-9]\.[0-9]\.[0-9])?$/, loader: "raw-loader"
    }, {
      test: /fonts\/.+\.(ttf|eot|svg)(\?v=[0-9]\.[0-9]\.[0-9])?$/, loader: "raw-loader"
    }, {
      test: /\.json$/, loader: "json-loader"
    }]
  }
};

var myRequire = require('enhanced-require')(module, options);

myRequire('./www');

Yep It’s not ideal - I don’t need loaders for png\jpg\woff\etc but I leaved them, cuz I have some calls to this access at client side (Work In Progress). They can be removed.