bazel: Bazel remote-caching does not properly treat volatile-status as if it never changes
Description of the problem / feature request:
When a target depends on the file volatile-status.txt
as an input, that target will not be rebuilt if volatile-status.txt
changes and the target is cached locally. However, the remote cache does not behave in the same way, and a change in the hash of volatile-status.txt
will cause the target not to read from the remote cache.
Bugs: what’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Clone the repo test_volatile_status and set up a bazel remote cache. Then run the following commands at the root of the repo:
bazel build --remote_cache=<your_remote_cache> //:artifact
bazel build --remote_cache=<your_remote_cache> //:artifact
# the above will read from the local cache, as expected
bazel clean
bazel build --remote_cache=<your_remote_cache> //:artifact
# this will not read from the remote cache
What operating system are you running Bazel on?
Ubuntu 16.04.6 LTS
What’s the output of bazel info release
?
release 0.29.0
What’s the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD
?
https://github.com/Panfilwk/test_volatile_status.git da002118b6c79bd7e33b017221c772fbfba4f009 da002118b6c79bd7e33b017221c772fbfba4f009
Have you found anything relevant by searching the web?
This issue touches on this concept for local caching, but nothing about how the behavior of this file works with remote caching.
Any other information, logs, or outputs that you want to share?
exec1.log.txt and exec2.log.txt are the resulting parsed experimental execution logs from the first and third build commands following the debug instructions here
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 29 (21 by maintainers)
Commits related to this issue
- Constant volatile status to improve cache hit rate (#80) bazel remote cache does not respect the intent of volatile status file, so let's ensure that the status variables are constant in our tests. ... — committed to grailbio/rules_r by deleted user 3 years ago
- bazel: avoid stamping --config=dev builds When used with a remote build cache, these stamps (despite being marked as volatile), induce an unnecessary golink step when switching back and forth between... — committed to irfansharif/cockroach by irfansharif 2 years ago
- bazel: disable sandboxing and avoid stamping - Sandboxing on MacOS have measurable overhead. See https://github.com/bazelbuild/bazel/issues/8230; it comes from symlinks being much slower to creat... — committed to irfansharif/cockroach by irfansharif 2 years ago
- bazel: disable sandboxing and avoid stamping - Sandboxing on MacOS have measurable overhead. See https://github.com/bazelbuild/bazel/issues/8230; it comes from symlinks being much slower to creat... — committed to irfansharif/cockroach by irfansharif 2 years ago
- bazel: disable sandboxing and avoid stamping - Sandboxing on MacOS have measurable overhead. See https://github.com/bazelbuild/bazel/issues/8230; it comes from symlinks being much slower to creat... — committed to irfansharif/cockroach by irfansharif 2 years ago
- Merge #79360 79360: bazel: disable sandboxing and avoid stamping r=irfansharif a=irfansharif - Sandboxing on MacOS have measurable overhead. See https://github.com/bazelbuild/bazel/issues/8230; i... — committed to cockroachdb/cockroach by deleted user 2 years ago
- build(bazel): improve remote caching for AIO local deps build Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt and vola... — committed to aspect-forks/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt and vola... — committed to aspect-forks/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt and vola... — committed to aspect-forks/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt and vola... — committed to aspect-forks/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt and vola... — committed to aspect-forks/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt and vola... — committed to aspect-forks/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build (#48579) Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt... — committed to angular/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build (#48579) Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt... — committed to angular/angular by kormide 2 years ago
- build(bazel): improve remote caching for AIO local deps build (#48579) Fix non-hermetic zipping of example zips by fixing the zip entry timestamps. I also hardcoded stamp values in stable-status.txt... — committed to trekladyone/angular by kormide 2 years ago
- Merge branch 'levsha/-IDX-2760-Pass-volatile-status-over-a-side-channel-' into 'master' [IDX-2760] Pass volatile status over a side channel. Passing it as dependency invalidates remote cache: https:... — committed to dfinity/ic by cgundy a year ago
We are also facing this issue, which impact the efficiency of our continuous delivery and load and fill, unnecessarily, our remote cache. I have just spent a little time to find the origin of the problem, and if it can help, here a minimal example to reproduce the behaviour.
Example
WORKSPACE
BUILD.bazel
myrule.bzl
A simple rule which copy the file
volatile-status.txt
tobazel-bin/dummy
. Exactly the same example than test_volatile_status.Scenarios
First build
We start from an empty environment. The Bazel local cache and the remote cache are fully cleaned.
bazel-bin/dummy
:bazel-out/volatile-status.txt
:As expected, the file, not present in the local cache and in the remote cache, is built and the remote cache is filled. We can see all executed gRPC functions from the log file:
Second build without clean cache
bazel-bin/dummy
:bazel-out/volatile-status.txt
:Not surprisingly, the file, already in the local cache is not rebuilt even if the volatile stamp file has changed. No gRPC requests are executed. This behaviour is conform with the documentation.
Third build with a clean of the local cache
bazel-bin/dummy
:bazel-out/volatile-status.txt
:The file, with a different
digestKey
than the first build, is searched from the remote cache and is not found. The file is then rebuilt and pushed again in the remote cache. We can see all similary executed gRPC requests than the first build.The computation of this
digestKey
should not take into account the updated content of thevolatile-status.txt
, as is the case when we remove or manually update the volatile stamp file between two builds.It’s the definition of the volatile file:
(from the user-manual 5.1.1)
This behaviour should be the same even if we use a remote cache.
Workaround
To avoid this issue, we are going to remove the use of the
--stamp
option and:If we can reopen this issue that would be really great and appreciated ❤️ !
For all who are still struggling with this issue, consider just making your volatile status files constant through your workspace status command. You can use an environment variable to control this behavior, which can be set or unset depending on whether you are testing or deploying. If the status files themselves are constant, you don’t have to worry about how they are handled by bazel and any language rules.
You can override bazel’s default values for status vars as in this example here.
I submitted some PR / issues to various rules (k8s, docker, etc.) that are affected by this to disable volatile status stamping but feels wrong to me. This really should be addressed here. I understand the concerns about correctness higher in the thread, but Bazel claims both Fast and Correct. This issue seriously impairs “Fast.”
Not sure I understand something. Wouldn’t the stamped output files be stored in the remote cache, and couldn’t bazel just ignore
volatile-status.txt
and just download the previously built output file?