bazel: Spuriously breakage in Gerrit CI after upgrading from 7.0.0rc2 to 7.0.0rc3
Description of the bug:
Gerrit Code Review is in process of upgrading to bazel 7.0.0.
All was fine after the upgrade to 7.0.0rc2, see: [1].
However, after upgrading to the 7.0.0rc3 we started to see this breakage on our CI:
https://gerrit-ci.gerritforge.com/job/Gerrit-verifier-chrome-latest/40214/console
INFO: Invocation ID: 93ef2f32-774e-40ce-b58d-24dd7a30b758
INFO: Options provided by the client:
Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/jenkins/workspace/Gerrit-verifier-chrome-latest/gerrit/.bazelrc:
'build' options: --noenable_bzlmod --workspace_status_command=python3 ./tools/workspace_status.py --repository_cache=~/.gerritcodereview/bazel-cache/repository --action_env=PATH --disk_cache=~/.gerritcodereview/bazel-cache/cas --java_language_version=17 --java_runtime_version=remotejdk_17 --tool_java_language_version=17 --tool_java_runtime_version=remotejdk_17 --incompatible_strict_action_env --announce_rc
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Analyzing: target //tools/maven:gen_api_install (1 packages loaded, 0 targets configured)
Analyzing: target //tools/maven:gen_api_install (1 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: Analyzed target //tools/maven:gen_api_install (1 packages loaded, 1 target configured).
[368 / 527] Executing genrule @jgit//org.eclipse.jgit:jgit; 1s remote-cache, linux-sandbox
[369 / 527] [Prepa] Compiling Java headers external/jgit/org.eclipse.jgit.ssh.apache/libssh-apache-hjar.jar (53 source files)
ERROR: /home/jenkins/workspace/Gerrit-verifier-chrome-latest/gerrit/java/com/google/gerrit/jgit/BUILD:3:13: Compiling Java headers java/com/google/gerrit/jgit/libjgit-hjar.jar (1 source file) failed: Failed to fetch blobs because they do not exist remotely.: Missing digest: cf3b2439c36619f2b6aaadddc55f15ddfd0c96566d22c1c507823ca74ac09732/127311204 for bazel-out/k8-fastbuild/bin/external/rules_java_builtin/toolchains/platformclasspath.jar
ERROR: /home/jenkins/workspace/Gerrit-verifier-chrome-latest/gerrit/java/com/google/gerrit/jgit/BUILD:3:13: Building java/com/google/gerrit/jgit/libjgit.jar (1 source file) failed: Failed to fetch blobs because they do not exist remotely.: Missing digest: cf3b2439c36619f2b6aaadddc55f15ddfd0c96566d22c1c507823ca74ac09732/127311204 for bazel-out/k8-fastbuild/bin/external/rules_java_builtin/toolchains/platformclasspath.jar
Target //tools/maven:gen_api_install failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.088s, Critical Path: 4.95s
INFO: 24 processes: 23 internal, 1 linux-sandbox.
ERROR: Build did NOT complete successfully
bazelisk failed to build gen_api_install. Use VERBOSE=1 for more info
Build step 'Execute shell' marked build as failure
Finished: FAILURE
If I downgrade to 7.0.0.rc2, then the build is successful again: [1]
[1] https://gerrit-review.googlesource.com/c/gerrit/+/391534
Which category does this issue belong to?
No response
What’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
I cannot currently reproduce the problem locally 😭
This command is invoked on the CI:
tools/maven/api.sh install
That is created a shell script and invoking it to publish Plugin API artifacts in the local maven repository.
Which operating system are you running Bazel on?
Linux
What is the output of bazel info release
?
7.0.0rc3
If bazel info release
returns development version
or (@non-git)
, tell us how you built Bazel.
No response
What’s the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD
?
No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
All is fine on Bazel 7.0.0.rc2. I am unable to reproduce the problem locally and this cannot bisect.
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
About this issue
- Original URL
- State: open
- Created 8 months ago
- Comments: 26 (20 by maintainers)
So, to verify, that remote cache contributes to the problem, I upgraded (again) the Bazel version from 7.0.0rc2 to 7.0.0rc3, and uploaded a new patch set (22). As explained in my previous comment, this would skip remote cache usage and the verification was successful: [1].
I’m going to remove the changes in
.bazelversion
and add the option--experimental_remote_cache_eviction_retries=5
, as suggested by @coeuvre .[1] https://gerrit-review.googlesource.com/c/gerrit/+/387837/22
I think they are using remote cache. The flag was passed with env:
Also,
xxx remote cache hit
indicates remote cache. For disk cache it would bexxx disk cache hit
.@bazel-io fork 7.0.0
@bazel-io flag