bullmq: [Bug]: ESM loader error. Sandbox processor not working

Version

4.15.2

Platform

NodeJS

What happened?

I’ve created a queue with a sandbox processor:

   const worker = new Worker(
      name,
      join(__dirname, name, `${name}.processor.js`),
      {
        connection: {
          host: this.configService.get('redisHost'),
          port: this.configService.get('redisPort'),
        },
        useWorkerThreads: true,
      },
    );

When I try to start the job I get the following error:

Error: Only URLs with a scheme in: file, data, and node are supported by the default ESM loader. On Windows, absolute paths must be valid file:// URLs. Received protocol 'c:'
    at throwIfUnsupportedURLScheme (node:internal/modules/esm/load:236:11)
    at defaultLoad (node:internal/modules/esm/load:128:3)
    at ModuleLoader.load (node:internal/modules/esm/loader:409:13)
    at ModuleLoader.moduleProvider (node:internal/modules/esm/loader:291:56)
    at new ModuleJob (node:internal/modules/esm/module_job:65:26)
    at ModuleLoader.#createModuleJob (node:internal/modules/esm/loader:303:17)
    at ModuleLoader.getJobFromResolveResult (node:internal/modules/esm/loader:260:34)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:241:17)
    at ModuleLoader.import (node:internal/modules/esm/loader:328:23)

Node: 20.1.0 OS: Windows 11 Typescript: 5.2.2

How to reproduce.

No response

Relevant log output

No response

Code of Conduct

I agree to follow this project’s Code of Conduct

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 20 (12 by maintainers)

Commits related to this issue

docs(sandbox): add URL support section (#2373) ref #2326 #2372 — committed to taskforcesh/bullmq by roggervalf 6 months ago

Most upvoted comments

Unfortunatelly this will not work for remote URLs, as neither fs.existsSync nor import support other than local files.

manast on Dec 21, 2023

The interesting thing about this change would be that the processor file could actually be an external URL… maybe useful after all.

manast on Dec 20, 2023

Hi @manast and @miroirstudiosrl

I found some problem with check file in Worker.ts source:

First of all, if in ‘windows’ system you try this:

const nodeurl = require('node:url');
const fs = require('fs');

const path = `${__dirname}\\index.js`;
const url = nodeurl.pathToFileURL(path);
const fileUrl = url.href;

console.log(path);
console.log(url);
console.log(fileUrl);

console.log(`with path, does file exist?: ${fs.existsSync(path)}`); // with path, does file exist?: true
console.log(`with url, does file exist?: ${fs.existsSync(url)}`); // with url, does file exist?: true
console.log(`with fileUrl, does file exist?: ${fs.existsSync(fileUrl)}`); // with fileUrl, does file exist?: false

you get this

URL {
  href: 'file:///C:/tmp/mono/index.js',
  origin: 'null',
  protocol: 'file:',
  username: '',
  password: '',
  host: '',
  hostname: '',
  port: '',
  pathname: '/C:/tmp/mono/index.js',
  search: '',
  searchParams: URLSearchParams {},
  hash: ''
}
file:///C:/tmp/mono/index.js
with path, file exists?: true
with url, file exists?: true
with fileUrl, file exists?: false

So the fs.existsSync function needs a URL object to evaluate true the existing file test. In the source of worker.ts

https://github.com/taskforcesh/bullmq/blob/67c3b8b7af266ba8be82fa1e095ac86c8b86e6a1/src/classes/worker.ts#L241

the program test a string. If you look at the costructor

https://github.com/taskforcesh/bullmq/blob/67c3b8b7af266ba8be82fa1e095ac86c8b86e6a1/src/classes/worker.ts#L196

the class needs a path file as string

I suspect that this is the problem, the program stop to work because it needs a string path file.

@manast can you change the costructor method of Worker class ?

x-etienne on Dec 20, 2023