nativelink: Upload never finishes (or takes a **very** long time)

I am currently trying to use nativelink 0.5.1 with buck2. It works fine when building our backend, but building the frontend on MacOS aarch64 never finishes since it is stuck in the re_upload phase:

[2024-08-30T09:32:37.761+00:00] Waiting on root//frontend:tsc_generated (prelude//platforms:default#200212f73efcd57d) -- action (genrule) [re_upload]
...

[2024-08-30T09:59:15.457+00:00] Waiting on root//frontend:tsc_generated (prelude//platforms:default#200212f73efcd57d) -- action (genrule) [re_upload]
...
[2024-08-30T10:05:15.465+00:00] Waiting on root//frontend:tsc_generated (prelude//platforms:default#200212f73efcd57d) -- action (genrule) [re_upload]

(it’s still running and I’ll see if it finishes at all; I’ll report back what happened)

I have started nativelink locally, using the basic config, just setting additional_environment.

In the build, there are these genrules:

genrule(
    name = "node_modules",
    srcs = [
        "package.json",
        "yarn.lock",
    ],
    out = "node_modules",
    cmd = "$(exe toolchains//:yarn) install && mv node_modules ../out",
    labels = [
        "yarn_install",  # causes this genrule to run locally (needs network access)
    ],
)

genrule(
    name = "tsc_generated",
    srcs = [
        ":generate_api_types",
        ":node_modules",
        "package.json",
        "tsconfig.json",
    ],
    out = "dist",
    cmd = """\
$(exe toolchains//:yarn) run tsc --outDir ${OUT} --rootDir . --declaration true
""",
    default_outs = ["dist"],
)

So, yes the node_modules output is indeed large:

$ fd . -tf buck-out/v2/gen/root/904931f735703749/frontend/__node_modules__/out/node_modules | wc -l
92822

node_modules is heavier than the universe

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 20 (6 by maintainers)

Commits related to this issue

Most upvoted comments

This issue was resolved with removing atime a while back, ahead of 0.6.0 here.

I started looking into this a couple days ago, but then had a conference to attend. I’ll try to get some time tomorrow/this-weekend to deep-dive this. These kind of things usually end up being a config problem somewhere, but I’ll keep you tuned.