bazel: Symlinked sandbox is slow
Description of the bug:
The symlinked sandbox is slow when there is a large number of input files (I have seen reports of actions with up to 300K)
There are a number of ways one could improve this:
- Creating the input directories and symlinks on multiple threads (
SandboxHelperscurrently does this on one thread) - Traversing the Java -> C++ boundary less frequently
- Using one symlink per large tree artifact instead of symlinking each file in it separately
- Using io_uring on Linux for more efficient data transfer to the kernel
- Keeping the file system created for an action around and re-using it if the same action (or a similar one) is executed again
What’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
No response
Which operating system are you running Bazel on?
No response
What is the output of bazel info release?
No response
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What’s the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 2
- Comments: 15 (12 by maintainers)
cc @larsrc-google
@lberki There is
--experimental_reuse_sandbox_directories- have you tried that?