knex: "Module not found" errors when using knex with webpack

also ran into this. what works for me;

externals: {
  knex: 'commonjs knex'
}

This issue seems to be just a problem how to setup webpack configuration when including knex… lets reopen if there are changes required in knex to support webpack better.

In case it helps, I’m porting a NextJS v12 project to v13 using app directories.

I was getting:

wait  - compiling /page (client and server)...
error - ./node_modules/knex/lib/dialects/better-sqlite3/index.js:7:11
Module not found: Can't resolve 'better-sqlite3'

Import trace for requested module:
./node_modules/knex/lib/dialects/index.js
./node_modules/knex/lib/knex-builder/internal/config-resolver.js
./node_modules/knex/lib/knex-builder/Knex.js
./node_modules/knex/lib/index.js
./node_modules/knex/knex.js
./lib/read-db.ts
./app/page.tsx
./node_modules/next/dist/build/webpack/loaders/next-app-loader.js?name=app%2Fpage&appPaths=%2Fpage&pagePath=private-next-app-dir%2Fpage.tsx&appDir=%2FUsers%2Fpeterbe%2Fdev%2FGITHUB%2FDOCS%2Factions-stats%2Fapp&pageExtensions=tsx&pageExtensions=ts&pageExtensions=jsx&pageExtensions=js&rootDir=%2FUsers%2Fpeterbe%2Fdev%2FGITHUB%2FDOCS%2Factions-stats&isDev=true&tsconfigPath=tsconfig.json!

https://nextjs.org/docs/messages/module-not-found

The solution was to add this to my next.config.js

  webpack: (
    config,
    { buildId, dev, isServer, defaultLoaders, nextRuntime, webpack }
  ) => {
    config.externals.push({
      knex: 'commonjs knex'
    })
    // Important: return the modified config
    return config
  },

Now, in app/page.tsx I can continue to use:

import knex from "knex";
...
async function getData() {
  const repos = await getReposWithStats();
  return repos
}
...

I can’t comment specifically on the mariasql problem but what solved my problem is to define all the drivers I don’t use as external. For instance, I use the mysql2 driver for my project so I have this in externals:

  externals: {
    // Possible drivers for knex - we'll ignore them
    'sqlite3': 'sqlite3',
    'mariasql': 'mariasql',
    'mssql': 'mssql',
    'mysql': 'mysql',
    'oracle': 'oracle',
    'strong-oracle': 'strong-oracle',
    'oracledb': 'oracledb',
    'pg': 'pg',
    'pg-query-stream': 'pg-query-stream'
  }

The result is that knex gets bundled without errors or warnings

@joeketchi Ignoring the node_modules directory doesn’t really solve the issue, we’re just working around the issue. It would be awesome if the code would do these requires in a more intelligent way that doesn’t throw errors.

Thanks for all the solutions so far! I tried adding the following to next.config.js and that seems to get rid of the issue as well if you don’t want to be creating your own webpack or pushing to config:

/** @type {import('next').NextConfig} */
const nextConfig = {
  experimental: {
    serverComponentsExternalPackages: ['knex'],
  },
};

module.exports = nextConfig;

downside is you get the experimental feature warning.

I had this issue, but adding a Regex to my externals in my webpack config fixed it:

const commonLoaders = [
  { test: /\.js$/, loader: 'babel', exclude: [/node_modules/] },
  { test: /\.json$/, loader: 'json' },
];

module.exports = [
  {
    name: 'browser',
    entry: './js/entry.js',
    output: {
      path: './build',
      filename: 'bundle.js',
    },
    module: {
      loaders: commonLoaders.concat([
        { test: /\.styl$/, loader: 'style!css!stylus' },
      ]),
    },
  },
  {
    name: 'server',
    entry: './server.js',
    target: 'node',
    output: {
      path: './build',
      filename: 'server.js',
      libraryTarget: 'commonjs2',
    },
    externals: [
      {
        'socket.io': 'socket.io',
      },
      /^[a-z\-0-9]+$/,
    ],
    module: {
      loaders: commonLoaders.concat([
        { test: /\.styl$/, loader: 'css!stylus' },
      ]),
    },
  },
];

I fixed it in webpack 5 using the IgnorePlugin.

const webpack = require('webpack');
//...snip
plugins: [
    new webpack.IgnorePlugin({
        resourceRegExp: /(sqlite3|pg|pg-query-stream|oracledb|mysql2|tedious)/
    }),
]

Hi folks, I was working to solve the problem of creating a complete webpack bundle for a service using knex and ran into this issue when trying to run migrations programmatically from that bundle. @mdlavin 's suggestion did get it to run the require but, it resulted in webpack not transforming the source file. It also meant I’d need to include that directory with the bundle out of band from the webpack bundle.

I figured out that you can use webpack’s ContextReplacementPlugin to resolve the issue with migrations and seeds. Below is a sample from my config (written in typescript):

plugins: [
  //...
  fs.existsSync(path.join(params.targetPackageFilePath, 'migrations')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/migrate/,
    '.',
    fs.readdirSync(path.join(params.targetPackageFilePath, 'migrations'))
      .reduce((result, file) => {
        return Object.assign(result, {
          [path.join(params.targetPackageFilePath, 'migrations', file)]: path.relative(
            path.dirname(require.resolve('knex/lib/migrate')),
            path.join(params.targetPackageFilePath, 'migrations', file)
          )
        });
      }, {})),

  fs.existsSync(path.join(params.targetPackageFilePath, 'seeds')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/seed/,
    '.',
    fs.readdirSync(path.join(params.targetPackageFilePath, 'seeds'))
      .reduce((result, file) => {
        return Object.assign(result, {
          [path.join(params.targetPackageFilePath, 'seeds', file)]: path.relative(
            path.dirname(require.resolve('knex/lib/seed')),
            path.join(params.targetPackageFilePath, 'seeds', file)
          )
        });
      }, {}))
  //...
]

In my config params.targetPackageFiletPath is the resolved path to the package being compiled e.g. /Users/yourusername/yourproject. So if a project at that path has migrations and seeds directories the resulting plugin configurations would look something like this:

  fs.existsSync(path.join(params.targetPackageFilePath, 'migrations')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/migrate/,
    '.',
    {
      '/Users/yourusername/yourproject/201706261234_migration.ts': '../../../migrations/201706261234_migration.ts'
    }),

  fs.existsSync(path.join(params.targetPackageFilePath, 'seeds')) && new webpack.ContextReplacementPlugin(
    /knex\/lib\/seed/,
    '.',
    {
      '/Users/yourusername/yourproject/seeds/201706261234_first-seed.ts': '../../../seeds/201706261234_first-seed.ts'
    }),

To break this down a little, the issue is that knex’s migration and seed code calls require to load your migrations and seeds, as it does here https://github.com/tgriesser/knex/blob/master/src/migrate/index.js#L54. It passes the result of a function/expression to require with no hard-coded string prefix or suffix so, when webpack attempts to figure out what files that might actually load it doesn’t have enough information to determine that. These plugins provide that needed information to webpack.

I’m not sure if Knex could be changed to make these unnecessary when bundling with webpack. Maybe by doing the requires in user-land?

Anyway, thought I’d share a solution that worked for me.

Actually, I don’t think the issue should be closed yet, there are still unresolved problems with webpack, with migration and seed:

WARNING in ./node_modules/knex/lib/migrate/index.js
342:20-81 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/migrate/index.js
447:18-49 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
113:13-74 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
150:11-24 Critical dependency: the request of a dependency is an expression

I couldn’t fix these

What I ended up doing was adding:

externals: ["knex", "pg"]

to webpack.config.js so that it doesn’t bundle them (adjust for the connector you’re using), and then doing:

npm --prefix ./dist install knex pg

Where prefix is the webpack output directory, right after npm run build (or webpack command)

2019/10 I managed to get it working, see my answer in another issue https://github.com/tgriesser/knex/issues/1446#issuecomment-537715431

@ifullgaz I found the suggestion from https://www.laurivan.com/make-electron-work-with-knex-js/ fixed these errors:

WARNING in ./node_modules/knex/lib/migrate/index.js
342:20-81 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/migrate/index.js
447:18-49 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
113:13-74 Critical dependency: the request of a dependency is an expression

WARNING in ./node_modules/knex/lib/seed/index.js
150:11-24 Critical dependency: the request of a dependency is an expression

Specifically, this part worked for me:

config.plugins = [
  // ...
  new NormalModuleReplacementPlugin(/\.\.\/migrate/, '../util/noop.js'),
  new NormalModuleReplacementPlugin(/\.\.\/seed/, '../util/noop.js'),
];

This worked great for me. I did this:

plugins: [
    new webpack.NormalModuleReplacementPlugin(/m[sy]sql2?|oracle(db)?|sqlite3/, "node-noop"),
    new webpack.NormalModuleReplacementPlugin(/\.\.\/migrate/, "node-noop"),
    new webpack.NormalModuleReplacementPlugin(/\.\.\/seed/, "node-noop")
  ]

1. I couldn’t get the relative import of noop.js to work because it was trying to import it deep inside knex. I just installed this simple node-noop lib and that worked fine.
2. We use Postgres, so the regex in the first line matches all the other ones it looks for. Modify to your heart’s content.

Any suggestion for this issue in Angular? I try using express project, it is normal (not showing any error). But when I use it in Angular project, it shows the error:

ERROR in ./node_modules/knex/lib/dialects/oracle/utils.js Module not found: Error: Can't resolve 'crypto' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib/dialects/oracle' ERROR in ./node_modules/knex/lib/dialects/mssql/index.js Module not found: Error: Can't resolve 'mssql/package.json' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib/dialects/mssql' ERROR in ./node_modules/knex/lib/runner.js Module not found: Error: Can't resolve 'stream' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib' ERROR in ./node_modules/knex/lib/dialects/oracledb/index.js Module not found: Error: Can't resolve 'stream' in '/Users/marudits/Documents/CODE/PROJECT/MOJOMARET/app-member/node_modules/knex/lib/dialects/oracledb'

where i can do the configuration to remove this error? could it be set up on angular.json or tsconfig.json file?

I looked a slightly different approach which allows webpack to process most require statements in the knex files, but will leave the non-literal requires alone so that the migration and seed files provided by the user of the library be found. Here is the approach I took:

config.module.rules = [
  {
    include: [
      /knex\/lib\/migrate\/index/,
      /knex\/lib\/seed\/index/
    ],
    loader: 'string-replace-loader',
    options: {
      search: 'require(\\([^\'"])',
      replace: '__non_webpack_require__$1',
      flags: 'g'
    }
  }
];