bazel-gazelle: Some go_repository gazelle generations are slow, could be pre-computed or cached

A common pathological case is @com_github_docker_docker which can spend hundreds of seconds generating BUILD files. Users are stuck waiting for this to run anytime the repository rule is invalidated.

@siggisim had the clever solution of checking in the resulting generated BUILD files rather than computing on-the-fly, using a script that converts to http_archive like https://github.com/buildbuddy-io/buildbuddy/blob/14ac4f6d9e3df578e31c129add910d176a63d075/deps.bzl#L876-L884

On Slack, @sluongng proposes the results are cachable: https://bazelbuild.slack.com/archives/CDBP88Z0D/p1647012317774269

naively speaking, a Gazelle cache is fairly easy to implement: Enable/disable by setting an environment variable pointing to a path Only cache if the go_repository SHA256 is set, use it as the cache key Under $HOME/.cache/gazelle, store the patch file under ./<sha256>.patch If cache not found, generate then store in cache

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 8
Comments: 18 (18 by maintainers)

Commits related to this issue

bazel: Upgrade gazelle Upgrading gazelle to the latest release fixes an issue where com_github_docker_docker takes many minutes to fetch (see https://github.com/bazelbuild/bazel-gazelle/issues/1190#i... — committed to minor-fixes/enkit by minor-fixes 2 years ago
bazel: Upgrade gazelle (#700) Upgrading gazelle to the latest release fixes an issue where com_github_docker_docker takes many minutes to fetch (see https://github.com/bazelbuild/bazel-gazelle/issu... — committed to enfabrica/enkit by minor-fixes 2 years ago

Most upvoted comments

@blico that’s fantastic, it did seem like just a needless de-optimization happening.

I tested your PR, get this timing locally:

# invalidate the go_repository, e.g. add `build_file_name = "BUILD.bazel,BUILD,invalidate1"`
% time bazel fetch @com_github_docker_docker//api
INFO: All external dependencies fetched successfully.
Loading: 20 packages loaded
bazel fetch @com_github_docker_docker//api  0.05s user 0.07s system 0% cpu 18.656 total

compared with the original network access de-opt

% time bazel fetch @com_github_docker_docker//api
Loading: 0 packages loaded
    Fetching @com_github_docker_docker; Running gazelle to generate BUILD files. Too slow? See https://github.com/bazelbuild/bazel-gazelle/issues/1190 159s
INFO: All external dependencies fetched successfully.
Loading: 1 packages loaded
bazel fetch @com_github_docker_docker//api  0.05s user 0.07s system 0% cpu 2:10.30 total

and compared with the checked-in patch file

% time bazel fetch @com_github_docker_docker//api
INFO: All external dependencies fetched successfully.
Loading: 1 packages loaded
bazel fetch @com_github_docker_docker//api  0.04s user 0.05s system 0% cpu 17.775 total

So I think you’ve totally solved this issue 🥳

alexeagle on Mar 16, 2022