bazel: Large number of inputs causes slowness / file handle exhaustion
Angular builds with Bazel, and uses npm to fetch dependencies. The number of files fetched is very large:
Projects/angular$ ls -R node_modules | wc -l
64545
The directory structure here has semantics that are relied on by nodejs module resolution logic, so we want to lay out this entire directory as an input to all nodejs rules. I observe a lot of time spent creating runfiles, and also MacOS users report they run out of file handles.
Proposal: a native rule similar to filegroup
except that it yields a TreeArtifact.
Note that this would potentially alleviate the problem in #374 - we wouldn’t need to reference files in this directory with labels.
@dslomov observes that this is tricky because TreeArtifacts have only been outputs in the past, so they don’t have any semantics for watching for changes, like source inputs have.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (14 by maintainers)
Update, we have a design doc to make rules_nodejs have smaller inputs by doing what pubref/rules_node does, while staying backwards compatible: https://docs.google.com/document/d/1AfjHMLVyE_vYwlHSK7k7yW_IIGppSxsQtPm9PTr1xEo/preview
For those of us following along from the outside world, resolve the go link, please.