bazel: Skyframe invalidation with disk or remote cache
Description of the bug:
When using a disk cache (possibly also a remote cache), Bazel repeatedly marks output files obtained from the cache as dirty to trigger a download, but then doesn’t download the files, so they are still missing on the next invocation.
Which category does this issue belong to?
No response
What’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
- Patch Bazel with the diff further below.
- Repeatedly run
baze --nosystem_rc --nohome_rc build --disk_cache=some_path //...on some project with Java targets.
See lines such as:
artifact is dirty: File:[[<execution_root>]bazel-out/darwin_arm64-fastbuild/bin]jeydtzir/libjeydtzir-hjar.jdeps RemoteFileArtifactValueWithExpiration{digest=0x7FACF6298A30C6CB373A988E3A3683CD3E8E9C46C59ECC570F089867234A3FEF, size=23, locationIndex=1, materializationExecPath=null, expireAtEpochMilli=1715713661059}
Invalidating: [ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=//vkprdlnk:vkprdlnk, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=1}, ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=//wztejdxc:wztejdxc, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=0}, ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=@@rules_java~//toolchains:platformclasspath, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=0}, ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=//jeydtzir:jeydtzir, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=1}, ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=//vkprdlnk:vkprdlnk, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=0}, ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=//mynbiqpm:mynbiqpm, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=1}, ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=//zjplsgqe:zjplsgqe, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=1}, ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=@@rules_java~//toolchains:platformclasspath, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=1}, ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=//jeydtzir:jeydtzir, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=0}, ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=//wztejdxc:wztejdxc, config=BuildConfigurationKey[b85621dae172f731976165fe86631f99030f401e70ed2b02d58c69595e5ea36d]}, actionIndex=1}] com.google.devtools.build.skyframe.InvalidatingNodeVisitor$DirtyingInvalidationState@104d38f9
Which operating system are you running Bazel on?
Linux
What is the output of bazel info release?
No response
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What’s the output of git remote get-url origin; git rev-parse HEAD ?
No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
diff --git a/src/main/java/com/google/devtools/build/lib/skyframe/FilesystemValueChecker.java b/src/main/java/com/google/devtools/build/lib/skyframe/FilesystemValueChecker.java
index 60a08ddf68..363c5291af 100644
--- a/src/main/java/com/google/devtools/build/lib/skyframe/FilesystemValueChecker.java
+++ b/src/main/java/com/google/devtools/build/lib/skyframe/FilesystemValueChecker.java
@@ -539,6 +539,7 @@ public class FilesystemValueChecker {
for (Map.Entry<Artifact, FileArtifactValue> entry : actionValue.getAllFileValues().entrySet()) {
if (artifactIsDirtyWithDirectSystemCalls(
knownModifiedOutputFiles, remoteArtifactChecker, entry, modifiedOutputsReceiver)) {
+ System.err.println("artifact is dirty: " + entry.getKey() + " " + entry.getValue());
isDirty = true;
}
}
@@ -551,6 +552,8 @@ public class FilesystemValueChecker {
tree.getChildValues().entrySet()) {
if (artifactIsDirtyWithDirectSystemCalls(
knownModifiedOutputFiles, remoteArtifactChecker, childEntry, modifiedOutputsReceiver)) {
+ System.err.println(
+ "tree artifact entry is dirty: " + childEntry.getKey() + " " + childEntry.getValue());
isDirty = true;
}
}
@@ -572,6 +575,7 @@ public class FilesystemValueChecker {
if (shouldCheckTreeArtifact(sortedKnownModifiedOutputFiles.get(), treeArtifact)
&& treeArtifactIsDirty(treeArtifact, entry.getValue())) {
// Count the changed directory as one "file".
+ System.err.println("tree artifact entry is dirty: " + treeArtifact);
modifiedOutputsReceiver.reportModifiedOutputFile(
getBestEffortModifiedTime(treeArtifact.getPath()), treeArtifact);
isDirty = true;
diff --git a/src/main/java/com/google/devtools/build/skyframe/EagerInvalidator.java b/src/main/java/com/google/devtools/build/skyframe/EagerInvalidator.java
index f8f2c0d560..07988108b3 100644
--- a/src/main/java/com/google/devtools/build/skyframe/EagerInvalidator.java
+++ b/src/main/java/com/google/devtools/build/skyframe/EagerInvalidator.java
@@ -82,6 +82,7 @@ public final class EagerInvalidator {
DirtyAndInflightTrackingProgressReceiver progressReceiver,
InvalidationState state)
throws InterruptedException {
+ System.err.println("Invalidating: " + diff + " " + state);
DirtyingNodeVisitor visitor =
createInvalidatingVisitorIfNeeded(graph, diff, progressReceiver, state);
if (visitor != null) {
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 16 (13 by maintainers)
Commits related to this issue
- Download runfiles in CompletionFunction. Previously, we download outputs of toplevel artifacts in CompletionFunction because intermediate targets might be promoted to toplevel by skymeld in the middl... — committed to bazelbuild/bazel by coeuvre 2 years ago
- Make incremental build with Skymeld and BwoB faster. Previously, a incremental build with Skymeld and BwoB has to re-evaluate action node in order to correctly staging toplevel artifacts. The reason ... — committed to bazelbuild/bazel by coeuvre 2 years ago
- Override LocalOutputService#getRemoteArtifactChecker to return RemoteArtifactChecker.IGNORE_ALL. Working towards #22367. PiperOrigin-RevId: 667965692 Change-Id: Ifc999a51d7ce4d5466d66c0b44e09e1491b8... — committed to bazelbuild/bazel by coeuvre 2 years ago
- Download runfiles in CompletionFunction. Previously, we download outputs of toplevel artifacts in CompletionFunction because intermediate targets might be promoted to toplevel by skymeld in the middl... — committed to coeuvre/bazel by coeuvre 2 years ago
- Download runfiles in CompletionFunction. Previously, we download outputs of toplevel artifacts in CompletionFunction because intermediate targets might be promoted to toplevel by skymeld in the middl... — committed to coeuvre/bazel by coeuvre 2 years ago
- Download runfiles in CompletionFunction. Previously, we download outputs of toplevel artifacts in CompletionFunction because intermediate targets might be promoted to toplevel by skymeld in the middl... — committed to coeuvre/bazel by coeuvre 2 years ago
- [7.4.0] Download runfiles in CompletionFunction. (#23442) Note that I have slightly modified the code to make it work with `release-7.4.0` source tree because of missing other commits. More specifica... — committed to bazelbuild/bazel by coeuvre 2 years ago
- Make incremental build with Skymeld and BwoB faster. Previously, a incremental build with Skymeld and BwoB has to re-evaluate action node in order to correctly staging toplevel artifacts. The reason ... — committed to coeuvre/bazel by coeuvre 2 years ago
- [7.4.0] Make incremental build with Skymeld and BwoB faster. (#23454) Previously, a incremental build with Skymeld and BwoB has to re-evaluate action node in order to correctly staging toplevel artif... — committed to bazelbuild/bazel by coeuvre 2 years ago
I have a proper fix and should be able to submit it in a few days.
Thanks! I’ll have to wait for an RC to start validation.
@bazel-io fork 7.4.0
Raising it to P2 but we probably won’t have time to work on the fix until next month or two. In the mean while, PR is welcome!