nativelink: Upload never finishes (or takes a **very** long time)
I am currently trying to use nativelink 0.5.1 with buck2. It works fine when building our backend, but building the frontend on MacOS aarch64 never finishes since it is stuck in the re_upload phase:
[2024-08-30T09:32:37.761+00:00] Waiting on root//frontend:tsc_generated (prelude//platforms:default#200212f73efcd57d) -- action (genrule) [re_upload]
...
[2024-08-30T09:59:15.457+00:00] Waiting on root//frontend:tsc_generated (prelude//platforms:default#200212f73efcd57d) -- action (genrule) [re_upload]
...
[2024-08-30T10:05:15.465+00:00] Waiting on root//frontend:tsc_generated (prelude//platforms:default#200212f73efcd57d) -- action (genrule) [re_upload]
(it’s still running and I’ll see if it finishes at all; I’ll report back what happened)
I have started nativelink locally, using the basic config, just setting additional_environment.
In the build, there are these genrules:
genrule(
name = "node_modules",
srcs = [
"package.json",
"yarn.lock",
],
out = "node_modules",
cmd = "$(exe toolchains//:yarn) install && mv node_modules ../out",
labels = [
"yarn_install", # causes this genrule to run locally (needs network access)
],
)
genrule(
name = "tsc_generated",
srcs = [
":generate_api_types",
":node_modules",
"package.json",
"tsconfig.json",
],
out = "dist",
cmd = """\
$(exe toolchains//:yarn) run tsc --outDir ${OUT} --rootDir . --declaration true
""",
default_outs = ["dist"],
)
So, yes the node_modules output is indeed large:
$ fd . -tf buck-out/v2/gen/root/904931f735703749/frontend/__node_modules__/out/node_modules | wc -l
92822

About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 20 (6 by maintainers)
Commits related to this issue
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
- [Breaking] Remove ResumableFileSlot and rely on high ulimits ResumableFileSlot became to difficult to manage, instead of managing resources this way, we use much higher DEFAULT_OPEN_FILE_PERMITS, set... — committed to allada/nativelink-fork by allada 2 years ago
This issue was resolved with removing atime a while back, ahead of
0.6.0here.I started looking into this a couple days ago, but then had a conference to attend. I’ll try to get some time tomorrow/this-weekend to deep-dive this. These kind of things usually end up being a config problem somewhere, but I’ll keep you tuned.