Report a bug

When attempting to benchmark types that rely on external dependencies, bench incorrectly returns 0 instantiations as the result.

Digging into this a bit further and logging out semantic diagnostics after this line reveals that the VFS is unable to resolve modules imported from the project’s node_modules, i.e. we get the “Cannot find module x or its corresponding type declarations.” diag message.

The root cause appears to be that the bench package creates the VFS using the in-memory System object here, which doesn’t have access to the node_modules in the current working directory.

If I use the file-system-backed System object instead, the issue is resolved.

const projectRoot = process.cwd();
const system = tsvfs.createFSBackedSystem(libFiles.defaultMapFromNodeModules, projectRoot, ts);

With this change I’m seeing the expected number of instantiations returned in the bench result.

If this change looks reasonable, I’d be happy to open a PR. If there are good reasons to use the in-memory System by default, perhaps we could make this configurable?

🔎 Search Terms

• bench cannot find module
• bench node_modules
• bench dependencies

🧩 Context

• ArkType version: @arktype/attest@0.1.1 - using attest separately.
• TypeScript version (4.8, 4.9, or 5.0): 5.2.2
• Other context you think may be relevant (Node version, OS, etc.): node@20

🧑‍💻 Repro

$ npm i -S @arktype/attest zod

// bench.ts
import { bench } from "@arktype/attest";
import { z } from "zod";

bench("zod bench", () => {
  const User = z.object({
    username: z.string(),
  });
  return {} as typeof User;;
}).types();

$ npx tsx bench.ts

Result:

🏌️ Result: 0instantiations
⛳ Baseline: 0instantiations

Expected result: ~18k instantiations