bazel: Bazel installer stopped working after d3f8efc, if project has .bazelversion file

We would like to kindly request this issue to be reopened as it does appears to have broken both Bazel and bazelisk at the same time in a significant manner without acceptable workarounds with the 2.0.0 release.

The requirements of CI and an organisation of 100+ engineers mean that the way in which we manage versions for bazel must be fully automated across all three supported platforms in a seamless way. This includes:

• Detecting what version of bazel needs to be used for a given invocation depending on configuration. The version might be different depending on the repository that is being built / tested.
• If the required version is not available on the current machine (local workstation, laptop, CI agent, …) then download it.
• Invoke the required version’s bazel binary with the original command and arguments passed to the top-level bazel invocation.

To achieve this:

• We have historically (3+ years) been using our in-house wrapper tools/bazel script to pin and auto-manage the versions required to build our various repositories. It provides nearly identical features to what bazelisk came to provide at a later point in time.
• In an effort to reduce tech-debt we want to move to simply use bazelisk as an off-the-shelf solution.

While attempting the move to bazelisk we are introducing the required .bazelversion file. While testing this on a Ubuntu workstation that has bazel 2.0.0 installed we found out that the new logic that was added to the wrapper script for bazel’s Debian package makes our attempt at migrating to bazelisk impossible.

The repo under tests currently relies on bazel 1.2.1. When running bazel version with our tools/bazel script stripped to simply invoke bazelisk, we would expect it to invoke bazelisk version which would:

1. Detect the required 1.2.1 version.
2. Detect that it is not currently available in bazelisk’s cache.
3. Download the 1.2.1 binary for Linux.
4. Invoke the freshly downloaded binary with the version command passed to bazelisk at the start.

However, instead, we found that we never even get intobazelisk or even tools/bazel and instead are presented with the already reported error message:

 -> bazel version
ERROR: The project you're trying to build requires Bazel 1.2.1 (specified in /<redacted>/.bazelversion), but it wasn't found in /usr/bin.

You can install the required Bazel version via apt:
  sudo apt update && sudo apt install bazel-1.2.1

If this doesn't work, check Bazel's installation instructions for help:
  https://docs.bazel.build/versions/master/install-ubuntu.html

Although the suggested solution is

to simply do what the (IMHO pretty helpful) error message suggests - install the required Bazel version

this manual actions is clearly not an option given the requirement of automating the entire process, which is the very goal of a tool like bazelisk, as well as Bazel itself.

From reading the code in the wrapper script we have also found that there is no way of deactivating this new logic.

As a result we are now faced with the fact that we cannot at any point in the foreseeable future migrate to bazelisk, even in the theoretical case if this feature was rolled back in a 3.0.0 release. It would still require a .bazelversion file and the mere presence of this file will trigger errors on any Debian-based machine that happens to have bazel 2.0.0 installed, since this feature does its checks before any other wrapper script or user configuration can intervene.

The questions that we have are:

1. What has been the reason to add this .bazelversion logic into the bazel wrapper script? It intermingles the functionality of bazel and bazelisk in an awkward way. The collision on the filename makes it virtually impossible to undo the implications of this change. Releasing a 3.0.0, or 2.0.1 for that matter is not sufficient as people can still have 2.0.0 installed.
2. Why is bazel trying to implement the features that bazelisk already provides in a clean and cross-platform way? The bazelisk tool is already part of the bazelbuild organisation / ecosystem on GitHub and can thus be clearly advertised for people needing the automated version management. All while leaving bazel itself do what it’s good at: building stuff in a fast and reproducible manner.
3. Why has this attempt actually been targeted at only one platform, resulting in bazelisk’s key-feature of automated version management not actually being supported all while actually breaking existing users of bazelisk on that platform?

To be clear: if we had already been using bazelisk before the release of 2.0.0 instead of our in-house script then the release of this “feature” would have actively broken all our Linux-based developers overnight.

Thanks for the input! I’ll go with @Helcaraxan’s plan and will make sure that the fixed wrapper (skip version check if tools/bazel exists) gets into Bazel 2.1.

Thanks everyone for commenting and/or upvoting @Helcaraxan’s post so that I can get a good feeling for the impact and urgency here and communicate this to my colleagues. This really helps prioritize this, so that we can hopefully find a good solution for all users quickly. 😃

Before Bazel 2.0, users could install Bazel using a variety of ways:

1. They could download the Bazel binaries directly (e.g. bazel-2.0.0-linux-x86_64) and check them into their source code repository or put them on their $PATH.
2. They could download the “installer script” (e.g. bazel-2.0.0-installer-linux-x86_64.sh) and install Bazel using that. This will actually install a shell-script as the bazel command, which tries to execute $REPO/tools/bazel if it exists, otherwise will execute Bazel.
3. They could install Bazel using package managers (apt-get, yum, brew, pacman, choco …). These packages also install the above mentioned shell-script.
4. They could install Bazelisk (usually as their bazel command on the PATH), which will automatically download the latest Bazel version (or the one specified in $REPO/.bazelversion), then check if a $REPO/tools/bazel script exists and if yes, run that, otherwise run the downloaded Bazel binary.

When the wrapper or Bazelisk run $REPO/tools/bazel they pass the location of the actual Bazel binary via the $BAZEL_REAL environment variable. The tools/bazel script is then expected to do whatever pre/post steps it wants to do (e.g. strip environment variables, run Bazel in a sandbox, upload logs after each invocation, …) and execute $BAZEL_REAL "$@" at the appropriate point in time.

All users from above list except the Bazelisk users would now regularly run into this problem: They somehow end up with a version of Bazel that doesn’t match the project that they want to build (e.g. downloaded the wrong version or their package manager automatically updated Bazel over night and whamm the entire team’s build is broken).

How do we fix that? Two ideas:

1. Ask people to check in Bazelisk as tools/bazel.
2. Change the wrapper script so that it understands .bazelversion and tells the user to install the correct version of Bazel using their native packaging solution (e.g. it will tell Debian users to apt-get install bazel-1.1.0, installer-script users to wget -O ... https://..., …).

Although I would have loved option 1, it didn’t sound too convincing in practice:

• The most important reason: Some people might not want to / are not allowed to / cannot download Bazel binaries from the internet in an automated way. For example, Google and other companies have IT security policies that prohibit downloading binaries from the internet as part of your build. (Yes, that means we can’t use Bazelisk even though I wrote it… 🤦‍♀) My previous company ran their entire CI without internet access for security reasons and to guarantee that e.g. “PyPI is down” or “Library foobar deleted their binaries from the internet” does not take the entire company down. These people need a way to version Bazel with their code that does not rely on Bazelisk downloading stuff from the internet.
• Bazelisk is a 6 MB binary per platform and if you check that in you could just as well check your Bazel binary in, right? Whereas if you don’t check-in Bazelisk but instead some script that gets Bazelisk for your current platform and then runs it - couldn’t your script just directly get Bazel instead?
• Checking in Bazelisk as tools/bazel doesn’t actually guarantee that it gets called - if your user is on Windows or installs Bazel by downloading Bazel binaries directly, their bazel invocation will not run tools/bazel.

So I went for option 2. The users who cannot use Bazelisk are happy that their Bazel versioning problem is now finally solved, yay. 🎉 The users who actually were using option 1 are now broken. Not-so-yay. 😞

Let’s figure out how to support this without breaking anyone. Here’s all combinations of users and repository setups. In the third column “Wrapper” refers to a script that simply changes the environment a bit and then delegates to $BAZEL_REAL, while “Bazelisk” refers to either Bazelisk or any other script that tries to find the correct Bazel version and then executes that (ignores $BAZEL_REAL).

My bazel is	$REPO/.bazelversion?	$REPO/tools/bazel?	Resolution
Binary	No	No	Not affected, works fine 🆗
Binary	Yes	No	Unsupported anyway 🤷‍♂
Binary	No	Wrapper	Unsupported anyway 🤷‍♂
Binary	Yes	Wrapper	Unsupported anyway 🤷‍♂
Binary	No	Bazelisk	Unsupported anyway 🤷‍♂
Binary	Yes	Bazelisk	Unsupported anyway 🤷‍♂
Shell-script	No	No	Not affected, works fine 🆗
Shell-script	Yes	No	Fixed by Bazel 2.0 🎉
Shell-script	No	Wrapper	Not affected, works fine 🆗
Shell-script	Yes	Wrapper	Fixed by Bazel 2.0 🎉
Shell-script	No	Bazelisk	Not affected, works fine 🆗
Shell-script	Yes	Bazelisk	Broken by Bazel 2.0 😭
Bazelisk	No	No	Not affected, works fine 🆗
Bazelisk	Yes	No	Not affected, works fine 🆗
Bazelisk	No	Wrapper	Not affected, works fine 🆗
Bazelisk	Yes	Wrapper	Not affected, works fine 🆗
Bazelisk	No	Bazelisk	Not affected, works fine 🆗
Bazelisk	Yes	Bazelisk	Not affected, works fine 🆗

It looks like we have to fix the case where:

• User installs Bazel using apt-get or similar that causes their bazel command to be Bazel’s shell script wrapper.
• Repository has a .bazelversion file.
• Repository has a tools/bazel script that is either Bazelisk or something similar.

Do you have suggestions how to fix this? Some ideas:

1. The shell-script wrapper could detect that tools/bazel is Bazelisk or a variant and then skip its own .bazelversion check. However I can’t immediately see a reliable way how to detect this.
2. We sacrifice the “Bazel is shell-script, repo has .bazelversion, tools/bazel is a simple pre-post-wrapper script” case and just always skip .bazelversion if tools/bazel exists. This might not be too bad - it would basically restore pre-2.0 behavior for this specific case only. I have to think about this a bit. 🤔
3. We completely remove the .bazelversion feature again from the shell-script wrapper and instead change Bazelisk so that it can be used by all users, even those in offline-only or security restricted scenarios. For example we could add a feature to Bazelisk so that it fetches its binaries and version-information from an internal location.
4. Other ideas?

1. What has been the reason to add this .bazelversion logic into the bazel wrapper script? It intermingles the functionality of bazel and bazelisk in an awkward way. The collision on the filename makes it virtually impossible to undo the implications of this change.

The reason was that users who could not use Bazelisk (or download binaries from the internet) still wanted a way to version Bazel with their source code (= the .bazelversion idea). The current solution makes it easy for users who install Bazel via apt-get to work with repositories that use .bazelversion files (but breaks the case where project owners assume that when a user runs bazel it will actually just run their $REPO/tools/bazel script).

Releasing a 3.0.0, or 2.0.1 for that matter is not sufficient as people can still have 2.0.0 installed.

I don’t see this quite as dark - if people installed Bazel 2.0.0 in a way that gave them the shell-script wrapper that you relied on, then it usually means that they used a package manager, which will automatically update their Bazel to later versions. Also, any Bazel version could potentially be so broken that it doesn’t work with your project - it’s not great, but we have to fix it and then go on.

1. Why is bazel trying to implement the features that bazelisk already provides in a clean and cross-platform way? The bazelisk tool is already part of the bazelbuild organisation / ecosystem on GitHub and can thus be clearly advertised for people needing the automated version management. All while leaving bazel itself do what it’s good at: building stuff in a fast and reproducible manner.

See above - I would love if everyone could just Bazelisk. Maybe I should have pushed more for this. If you and others think that adding offline support etc. to Bazelisk is the right direction, we can try to do that instead.

1. Why has this attempt actually been targeted at only one platform, resulting in bazelisk’s key-feature of automated version management not actually being supported all while actually breaking existing users of bazelisk on that platform?

I don’t think this is the case - all platforms that ship the shell-script wrapper (which are all platforms for which a tools/bazel approach would have worked) now support the .bazelversion file, too. Bazelisk supports more platforms, which is why I recommend users to directly install Bazelisk as their bazel.

If the question is about “why didn’t you just add this directly to Bazel” - because I didn’t want to add something that “phones home” to the internet to Bazel. User trust is super important for me and others on the Bazel team and we think your build tool shouldn’t rely on the internet in order to work.

Hope this already answers some questions - I’ll follow-up tomorrow in case I have any other ideas / thoughts about this and of course for any comments and questions that come up 😃

Just wanted to follow up @philwo about the resulting fix that now is making its way into 2.1.0. The reactiveness and communication has been greatly appreciated.

A great thanks on behalf of the entire team. ❤️

Is this a release blocker for 2.0?

Great resolution @philwo! Do we want to have a 2.0 patch release for this?

@philwo Thanks for adding the option to skip the check in the script wrapper.

Now, to test a minor Bazel upgrade, say 2.1.0rc4 from [1] in Gerrit workspace, where .bazelversion contains 2.0.0, I could add this tool/bazel to skip the check:

  $ cd gerrit
  $ cat tools/bazel
echo ${BAZEL_REAL}
$HOME/.bazel/bin/bazel-real "$@"

Now, after installing bash bazel-2.1.0rc4-installer-linux-x86_64.sh --user locally, I can actually test it, and the check is skipped:

  $ bazel version
/home/davido/.bazel/bin/bazel-2.0.0
Starting local Bazel server and connecting to it...
Build label: 2.1.0rc4

Of course, I could temporary update the .bazelversion in the tree, temporary remove it, or just pass USE_BAZEL_VERSION=2.1.0rc4 bazel version…

[1] https://releases.bazel.build/2.1.0/rc4/index.html

Hi @Helcaraxan and others,

sorry for the trouble you and others are having with the new feature. I’ll respond tomorrow to all of your questions. I designed the new feature to not break any existing users, but obviously this didn’t work out. I reopened this issue - let’s see that we can fix it together to also work for you and others who check in Bazelisk as tools/bazel into their repo.

Also, I don’t understand this reasoning:

However, all projects that have .bazelversion stopped working after d3f8efc. So the workaround for now is to remove .bazelversion file from the tree.

No, of course not - the workaround is not to remove the .bazelversion file, the “workaround” is to simply do what the (IMHO pretty helpful) error message suggests - install the required Bazel version 😃 I don’t see how this is “broken” or how “Bazel stopped working”.

Edit: @davido Sorry, to be clear, I totally understand why having to download the same Bazel version twice (once as the “latest” Bazel and once as the “specifically versioned package” Bazel) is weird and not a good UX. I’ll fix this before we release Bazel 2.0. The general behavior should be fine and an improvement, though, or what do you think?

So, here are some thoughts on this:

If a project has a .bazelversion file, it means it “should” be built with that exact Bazel version. Because the .bazelversion file is used to basically “version your build tool with your source code”, I do not want to implement fuzzy matching like “as long as it’s the same major version, it’s fine”. It will use and require the exact version, so this is working as intended.

Let’s assume you have a project with a .bazelversion file that specifies “1.1.0” as the required Bazel version, then here’s what happens:

• If you use Bazelisk, it will automatically and transparently use the correct version.
• If you installed Bazel via apt-get and you don’t have the bazel-1.1.0 package installed, it will suggest to run the command to install that package.
• If you install Bazel via the installer, it will suggest the command to download Bazel 1.1.0 and put it into the correct place.
• If you use a plain Bazel binary (no wrapper script), it will ignore the .bazelversion file and just use whatever Bazel version you have.

I think this behavior makes sense and is pretty user friendly?

However, I agree that there’s one optimization we should add: If the .bazelversion asks for Bazel X.Y and the Bazel version that was installed via apt-get install bazel or the installer script is already Bazel X.Y, then we do not need to download the same version once again. (Basically what Yun suggested.)

I’ll think of some logic for the wrapper script that supports this, then we can do a cherry-pick and it should be fine.