bazel: Java crashes due to hsperfdata file conflicts across sandboxes

Any updates on this? I started hitting this issue very often on all linux machines in CI.

My current workaround is to add the following to my .bazelrc

build --enable_platform_specific_config

build:linux --sandbox_tmpfs_path=/tmp

Yeah, in fact, I have to stop myself from campaigning for the flag flip to go into Bazel 6.0 😃

Emotionally, I feel like “Bazel can’t reliably run a JVM in an action” is quite an embarrassing issue, although the fact that this bug has been open for more than five years seems to imply that it’s a less serious issue than my feelings say.

Interestingly, HotSpot recently fixed this problem, too: openjdk/jdk@84f2314

Unfortunately fixing the problem was accompanied by a warning message printed to STDOUT, which is breaking some of our build actions that write to STDOUT (https://github.com/google/google-java-format), and filling our build logs with hundreds of those warnings for all other JVM-tool actions. I’m not sure how we were not affected by the crash, but are now affected by the logging, but hopefully --incompatible_sandbox_hermetic_tmp will fix that new problem for us. We’re still on Bazel 5.3, but we’ll be sure to try this flag when we can.

Update: now running the JVM in the sandbox should be stable with the --incompatible_hermetic_sandbox_tmp command line option with Bazel@HEAD (after 8e32f44)

I’d appreciate if you gave it a try; we are planning to flip that flag eventually (right, @larsrc-google ?) and thus the more testing and in the more diverse environment, the better.

@philwo thanks, now I get it; for some reason I thought that the sandbox individually bind mounts the input files instead of symlinking.

I’d take special-casing Java in the sandbox code over Java randomly crashing in actions any day; that way, at least it’s only us who get to see the ugliness.

I was wondering if we could get away with a more-complicated-than-seems-necessary solution:

• We bind mount the workspace, bazel-out/ and $OUTPUT_BASE/external/ to well-known locations in the sandbox (/bazel-workspace, /bazel-out, /bazel-external or something, doesn’t really matter)
• We direct the symlinks in the sandbox to those directories and not to their “real” locations
• We bind mount /tmp to an empty directory as above

Then the sandbox would see consistent output paths, it would work if the output base or the workspace is under /tmp and it wouldn’t clash with anything on the “real” file system except with these /bazel-* paths if someone is mad enough to have those on their file system or maybe in nested Bazel invocations (but even then, one could add a unique string per action to the path)

However, IIRC runfiles trees contain absolute symlinks to their contents, so they would break if they are symlinked “naively”. It’s not an unsolvable problem because the remote execution strategy solves it, but it does require some extra thinking.

Why do we have a PID namespace in the sandbox?

The reason why this bug exists is that actions have separate PID namespaces but a mostly shared file system, which strikes me as odd: we either try to isolate actions as fully as possible (but then how come they share /tmp?) or we only try to protect against mostly-accidental hermeticity violations (but then why the PID namespace?)