bazel: Spuriously breakage in Gerrit CI after upgrading from 7.0.0rc2 to 7.0.0rc3

Description of the bug:

Gerrit Code Review is in process of upgrading to bazel 7.0.0.

All was fine after the upgrade to 7.0.0rc2, see: [1].

However, after upgrading to the 7.0.0rc3 we started to see this breakage on our CI:

https://gerrit-ci.gerritforge.com/job/Gerrit-verifier-chrome-latest/40214/console

INFO: Invocation ID: 93ef2f32-774e-40ce-b58d-24dd7a30b758
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /home/jenkins/workspace/Gerrit-verifier-chrome-latest/gerrit/.bazelrc:
  'build' options: --noenable_bzlmod --workspace_status_command=python3 ./tools/workspace_status.py --repository_cache=~/.gerritcodereview/bazel-cache/repository --action_env=PATH --disk_cache=~/.gerritcodereview/bazel-cache/cas --java_language_version=17 --java_runtime_version=remotejdk_17 --tool_java_language_version=17 --tool_java_runtime_version=remotejdk_17 --incompatible_strict_action_env --announce_rc
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Analyzing: target //tools/maven:gen_api_install (1 packages loaded, 0 targets configured)
Analyzing: target //tools/maven:gen_api_install (1 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: Analyzed target //tools/maven:gen_api_install (1 packages loaded, 1 target configured).
[368 / 527] Executing genrule @jgit//org.eclipse.jgit:jgit; 1s remote-cache, linux-sandbox
[369 / 527] [Prepa] Compiling Java headers external/jgit/org.eclipse.jgit.ssh.apache/libssh-apache-hjar.jar (53 source files)
ERROR: /home/jenkins/workspace/Gerrit-verifier-chrome-latest/gerrit/java/com/google/gerrit/jgit/BUILD:3:13: Compiling Java headers java/com/google/gerrit/jgit/libjgit-hjar.jar (1 source file) failed: Failed to fetch blobs because they do not exist remotely.: Missing digest: cf3b2439c36619f2b6aaadddc55f15ddfd0c96566d22c1c507823ca74ac09732/127311204 for bazel-out/k8-fastbuild/bin/external/rules_java_builtin/toolchains/platformclasspath.jar
ERROR: /home/jenkins/workspace/Gerrit-verifier-chrome-latest/gerrit/java/com/google/gerrit/jgit/BUILD:3:13: Building java/com/google/gerrit/jgit/libjgit.jar (1 source file) failed: Failed to fetch blobs because they do not exist remotely.: Missing digest: cf3b2439c36619f2b6aaadddc55f15ddfd0c96566d22c1c507823ca74ac09732/127311204 for bazel-out/k8-fastbuild/bin/external/rules_java_builtin/toolchains/platformclasspath.jar
Target //tools/maven:gen_api_install failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 6.088s, Critical Path: 4.95s
INFO: 24 processes: 23 internal, 1 linux-sandbox.
ERROR: Build did NOT complete successfully
bazelisk failed to build gen_api_install. Use VERBOSE=1 for more info
Build step 'Execute shell' marked build as failure
Finished: FAILURE

If I downgrade to 7.0.0.rc2, then the build is successful again: [1]

[1] https://gerrit-review.googlesource.com/c/gerrit/+/391534

Which category does this issue belong to?

No response

What’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I cannot currently reproduce the problem locally 😭

This command is invoked on the CI:

  tools/maven/api.sh install

That is created a shell script and invoking it to publish Plugin API artifacts in the local maven repository.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

7.0.0rc3

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What’s the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

All is fine on Bazel 7.0.0.rc2. I am unable to reproduce the problem locally and this cannot bisect.

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

About this issue

  • Original URL
  • State: open
  • Created 8 months ago
  • Comments: 26 (20 by maintainers)

Commits related to this issue

Most upvoted comments

So, to verify, that remote cache contributes to the problem, I upgraded (again) the Bazel version from 7.0.0rc2 to 7.0.0rc3, and uploaded a new patch set (22). As explained in my previous comment, this would skip remote cache usage and the verification was successful: [1].

I’m going to remove the changes in .bazelversion and add the option --experimental_remote_cache_eviction_retries=5, as suggested by @coeuvre .

[1] https://gerrit-review.googlesource.com/c/gerrit/+/387837/22

I think they are using remote cache. The flag was passed with env:

[EnvInject] - Injecting as environment variables the properties content 
BAZEL_OPTS=--remote_cache=https://gerrit-ci.gerritforge.com/cache

Also, xxx remote cache hit indicates remote cache. For disk cache it would be xxx disk cache hit.

@bazel-io fork 7.0.0