bazel: Large number of inputs causes slowness / file handle exhaustion

Angular builds with Bazel, and uses npm to fetch dependencies. The number of files fetched is very large:

Projects/angular$ ls -R node_modules | wc -l
64545

The directory structure here has semantics that are relied on by nodejs module resolution logic, so we want to lay out this entire directory as an input to all nodejs rules. I observe a lot of time spent creating runfiles, and also MacOS users report they run out of file handles.

Proposal: a native rule similar to filegroup except that it yields a TreeArtifact. Note that this would potentially alleviate the problem in #374 - we wouldn’t need to reference files in this directory with labels. @dslomov observes that this is tricky because TreeArtifacts have only been outputs in the past, so they don’t have any semantics for watching for changes, like source inputs have.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 16 (14 by maintainers)

Most upvoted comments

Update, we have a design doc to make rules_nodejs have smaller inputs by doing what pubref/rules_node does, while staying backwards compatible: https://docs.google.com/document/d/1AfjHMLVyE_vYwlHSK7k7yW_IIGppSxsQtPm9PTr1xEo/preview

This was done in May 2017, see go/bazel-node-rules

For those of us following along from the outside world, resolve the go link, please.