bazel: "unlinkat ([path]) (Directory not empty)" errors with experimental_worker_for_repo_fetching

Description of the bug:

After upgrading to Bazel 7.2.0 and removing --experimental_worker_for_repo_fetching=off from .bazelrc (because https://github.com/bazelbuild/bazel/issues/21803 was fixed), we started to observe failures of the following form:

ERROR: /home/circleci/figma/WORKSPACE.bazel:157:27: fetching node_repositories rule //external:nodejs_linux_amd64: java.io.IOException: unlinkat (/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/nodejs_linux_amd64/bin/nodejs) (Directory not empty)
ERROR: no such package '@@nodejs_linux_amd64//': unlinkat (/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/nodejs_linux_amd64/bin/nodejs) (Directory not empty)

These errors appear to be linked with repository fetch restarts, particularly “fetch interrupted due to memory pressure; restarting”. In two of two cases where I have seen this error, it happened in a repo where the repo fetch got restarted, judging by either the presence of that log message or the JSON trace profile.

Which category does this issue belong to?

No response

What’s the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 7.2.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What’s the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

Bazel Slack discussion: https://bazelbuild.slack.com/archives/CA31HN1T3/p1718057176146129

Any other information, logs, or outputs that you want to share?

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 24 (16 by maintainers)

Commits related to this issue

Most upvoted comments

In case any other mac users happen across this issue, you can resolve it by disabling VirtioFS on the current Docker Desktop for Mac release (4.40.0 at the time of writing) and switching to osxfs “legacy” as shown here:

Image

We used to see concurrent write issues on Windows because of pyc creation happening, but fixed it. This would happen during the build phase, not repo phase.

Part of how the PYC creation process works is Python writes foo.pyc.NNNNN, where N is a timestamp. Then it does mv(foo.pyc.NNN, foo.pyc) to perform an atomic rename. This would happen after it created the pycache directory and within that directory (assuming it could; if it can’t, it doesn’t try to precompile)

To fix this, we:

  1. The globs ignore pyc files and pycache directories

  2. The repo files/dirs are all made read-only

On Windows, if a user is an admin, (2) doesn’t actually work. Hence (1) is the defense in the case.

On Mac/Linux, we never saw issues.

Hope this helps

On Thu, Jul 25, 2024, 9:43 AM Javier Maestro @.***> wrote:

@fmeum https://github.com/fmeum BTW, just found this while looking for the implementation of deleteTreesBelow:

https://github.com/bazelbuild/bazel/blob/d915b98376295bd010f669e7a60815a3eeca3d0c/src/main/native/unix_jni.cc#L1075-L1089

Part of the comment was added by @oquenchil https://github.com/oquenchil in d6c79db https://github.com/bazelbuild/bazel/commit/d6c79dbde3365c2e0e370eba4942ea49e3533395 and in the commit it was mentioned:

Asynchronous deletion may make it more likely for the problem to surface but the likelihood is still very low.

Could this be the issue? Maybe running the deletes in a Docker volume running in MacOS makes triggering the bug more likely :-?

— Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/22680#issuecomment-2250950723, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIEXQ6RKFPUBALRE6CGB7PLZOETKHAVCNFSM6AAAAABJDGQYW2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJQHE2TANZSGM . You are receiving this because you were mentioned.Message ID: @.***>

CC @rickeylev @aignas in case you know of any logic in the rules_python repo rules that could result in concurrent access to the files it deletes.

Since the stack trace is produced by an rctx.delete call in a Python repo rule and the crash still happens with virtual workers turned off, I wonder whether this may be a legitimate domain-specific non-Bazel issue. Have you raised this on the rules_python repo? I won’t rule out that this is ultimately a Bazel issue though.

@bazel-io fork 7.2.1