vite: Cannot bundle .node files

Describe the bug

I am trying to create a server side bundle that includes packages containing .node files. In this specific instance, the problematic package is nodejs-polars.

in/nodejs-polars.linux-arm-gnueabihf.node does not exist
✓ 13 modules transformed.
✓ built in 1.39s
[commonjs] Unexpected character '' (Note that you need plugins to import files that are not JavaScript)
file: /home/projects/vitejs-vite-xp1ggj/node_modules/nodejs-polars/bin/series/list.js:1:0
1: ELF>�rd@X�@8
   ^
2: @@@0�bd�bd...
3: 

I have tried various plugins to resolve this but none of them seem to work.

My config is as follows:

import native from 'vite-plugin-native';
export default function () {
  return {
    ssr: {
      noExternal: ['nodejs-polars'],
    },
    optimizeDeps: {
      include: ['linked-dep', 'node_modules'],
      exclude: ['nodejs-polars'],
    },
    esbuild: {
      platform: 'node',
      target: 'node16',
    },
    build: {
      lib: {
        entry: './main.js',
        name: 'demo',
      },
      ssr: true,
      target: 'node16',
    },
    plugins: [native({ target: 'esm' })],
  };
}

Reproduction

https://stackblitz.com/edit/vitejs-vite-xp1ggj?file=vite.config.ts,package.json&terminal=dev

Steps to reproduce

Run vite build --config vite.config.ts

System Info

System:
    OS: Linux 5.4 Ubuntu 20.04.6 LTS (Focal Fossa)
    CPU: (8) x64 AMD FX(tm)-8350 Eight-Core Processor
    Memory: 9.66 GB / 31.31 GB
    Container: Yes
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 18.14.2 - ~/.nvm/versions/node/v18.14.2/bin/node
    Yarn: 1.22.19 - ~/.nvm/versions/node/v18.14.2/bin/yarn
    npm: 8.9.0 - ~/.nvm/versions/node/v18.14.2/bin/npm
  Browsers:
    Brave Browser: 115.1.56.14
    Chrome: 116.0.5845.96



### Used Package Manager

npm

### Logs

_No response_

### Validations

- [X] Follow our [Code of Conduct](https://github.com/vitejs/vite/blob/main/CODE_OF_CONDUCT.md)
- [X] Read the [Contributing Guidelines](https://github.com/vitejs/vite/blob/main/CONTRIBUTING.md).
- [X] Read the [docs](https://vitejs.dev/guide).
- [X] Check that there isn't [already an issue](https://github.com/vitejs/vite/issues) that reports the same bug to avoid creating a duplicate.
- [X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to [vuejs/core](https://github.com/vuejs/core) instead.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitejs/vite/discussions) or join our [Discord Chat Server](https://chat.vitejs.dev/).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 16
  • Comments: 17

Most upvoted comments

We’ve managed to get .node files working, see our blog post here

The handling of .node files (and other binary-like files) which occur in nodejs packages needs to be addressed by either vite or a plugin. If vite will not support this (as it could be considered out of scope) then it will have to be a plugin. Our business interests have moved away from this at the moment, so making a plugin would be very low priority for @forgetso and myself at Prosopo.

Further, there’s more complications with files other than .node files (iirc .glyph) which serve similar purpose, so we’d need to do a bunch of research into how to handle those files correctly.

tldr; probably better for someone who knows more about these files to implement a plugin / build support in vite

It looks like an even simpler version also works:

{
      name: 'require-node-binaries-plugin',
      transform: (code, id) => {
        if (id.endsWith('.node')) {
          return code.replace(/export default "(.+)"/, 'export default require(".$1")');
        }

        return;
      },
    }

paired with build.ssrEmitAssets: true and assetsInclude: ['**/*.node']

Well, that’s one hell of a hack. But I’ll have to study and use it I guess. Thank you so much. Resolving .node files should be a feature in Vite.

I managed to bundle nodejs-polars using this config but it’s incredibly hacky.

I have many native deps and its works fine, but sqlite3 / better-sqlite3 have some different code to load .node files.

In my case, fully solved by "rollup-plugin-natives": "^0.7.8"

Image

For those searching for a solution, I (chatgpt to be honest) came up with a quick plugin:

function nativeFilesPlugin(): PluginOption {
  const files = new Map<string, { readonly fileName: string; readonly fileContent: Buffer }>();

  return {
    name: 'node-binaries-plugin',
    async load(id) {
      if (!id.endsWith('.node')) {
        return null;
      }

      const fileContent = await fs.readFile(id);
      const hash = createHash('sha256').update(fileContent).digest('hex').slice(0, 8);
      const fileName = `${path.basename(id, '.node')}.${hash}.node`;
      files.set(id, { fileName, fileContent });

      return `export default require('./${fileName}');`;
    },

    generateBundle(_, bundle) {
      for (const [id, { fileName, fileContent }] of files.entries()) {
        this.emitFile({ type: 'asset', fileName, source: fileContent });
        delete bundle[id];
      }
    },
  };
}

I’m building into cjs, so I also have these settings:

build: {
  rollupOptions: {
    output: {
      format: 'cjs',
    },
  },
  commonjsOptions: {
    requireReturnsDefault: 'auto',
  }
}