esbuild: Racey `Could not resolve file` errors despite files existing

When running a working esbuild config in a new environment, I get frequent but inconsistent Could not resolve <file> errors despite the files in question existing just fine.

I’m trying to get an esbuild configuration that works locally on my OS X machine and in a Github Actions Ubuntu runner onto another Linux based CI system. The exact same config works fine always in the other environments, but only sometimes works in the new environment. I am building a big TypeScript project for node by passing ~200 or so .ts entrypoint files to the JS build API.

The files that esbuild fails to resolve change each time, but each time it does fail, it’s all of the files in a particular folder that fail to resolve:

error: Could not resolve "/app/packages/api/src/routes/edit/graphql/1_scalars.ts"
error: Could not resolve "/app/packages/api/src/routes/edit/graphql/connections.ts"
error: Could not resolve "/app/packages/api/src/routes/edit/graphql/scope.ts"
error: Could not resolve "/app/packages/api/src/routes/edit/graphql/auth.ts"
error: Could not resolve "/app/packages/api/src/routes/edit/graphql/liveTreeSchema.ts"

I don’t have any resolve plugins set up and I am running on v0.11.1.

Debugging

I tried adding a resolve plugin that runs an fs.promises.access on each file and then returns undefined to prove that the same process invoking esbuild can access the file fine, and it indeed logged successful accesses. Because of this and because the build works sometimes I think the files really are there.

Curiously, strace-ing esbuild makes it fail to resolve some files almost every time I invoke it with strce. I don’t see anything amiss in the strace, but here’s an example trace for a file that fails to resolve:

4801  openat(AT_FDCWD, "/app/packages/api/src/services/app-graphql-api/meta/GadgetMetaSchema.ts", O_RDONLY|O_CLOEXEC) = 6
4801  epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2300782936, u64=140580875348312}} <unfinished ...>
4801  <... epoll_ctl resumed> )         = 0
4801  fcntl(6, F_GETFL)                 = 0x8000 (flags O_RDONLY|O_LARGEFILE)
4801  fcntl(6, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0
4801  getdents64(6, 0xc0005da000, 8192) = -1 ENOTDIR (Not a directory)
4801  epoll_ctl(3, EPOLL_CTL_DEL, 6, 0xc00019296c <unfinished ...>
4801  <... epoll_ctl resumed> )         = 0
4801  close(6)                          = 0
4801  write(2, "\33[1m > \33[31merror: \33[0m\33[1mCould"..., 124) = 124
4801  newfstatat(AT_FDCWD, "/app/packages/api/src/services/live-tree-server/listeners", {st_mode=S_IFDIR|0775, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0

To me it seems like opening the file succeeds just fine, and then esbuild writes the error: Could ... string to stderr.

I was running strace -f -e '!futex,read' yarn <my-esbuild-task> which ends up running esbuild with these options:

{
        entryPoints: [...<filenames>],
        incremental: true,
        bundle: false,
        platform: "node",
        format: "cjs",
        target: ["node14"],
        outdir: <dir>,
        outbase: <dir>,
        sourcemap: "inline",
}

see https://github.com/gadget-inc/esbuild-dev/blob/main/src/Compiler.ts#L98-L106 for the source

For contrast, here’s what the strace looks like for esbuild reading a file successfully early:

4801  newfstatat(AT_FDCWD, "/app/tsconfig.json", {st_mode=S_IFREG|0664, st_size=422, ...}, AT_SYMLINK_NOFOLLOW) = 0
4801  newfstatat(AT_FDCWD, "/app/tsconfig.json",  <unfinished ...>
4801  <... newfstatat resumed> {st_mode=S_IFREG|0664, st_size=422, ...}, 0) = 0
4801  openat(AT_FDCWD, "/app/tsconfig.json", O_RDONLY|O_CLOEXEC <unfinished ...>
4801  <... openat resumed> )            = 6
4801  epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2300782936, u64=140580875348312}} <unfinished ...>
4801  <... epoll_ctl resumed> )         = 0
4801  fcntl(6, F_GETFL)                 = 0x8000 (flags O_RDONLY|O_LARGEFILE)
4801  fcntl(6, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE <unfinished ...>
4801  <... fcntl resumed> )             = 0
4801  fstat(6, {st_mode=S_IFREG|0664, st_size=422, ...}) = 0
4801  epoll_ctl(3, EPOLL_CTL_DEL, 6, 0xc00013024c <unfinished ...>
4801  <... epoll_ctl resumed> )         = 0
4801  close(6 <unfinished ...>
4801  <... close resumed> )             = 0

I realize this isn’t a super reproducible issue, but I am hoping that these are enough breadcrumbs that we could figure out what next steps to take to unearth more information. Also notable is that this seems similar to #348 , but ulimit -n is 65536 in this environment, and I am running as root.

Any other information I could gather to help figure out what might be causing this heisenbug?