setup-python: brew update followed by brew upgrade fails for Python's 2to3 conflict (December 2022)

Any chance to have it fixed in the image? All was working before, I wouldn’t want to use any workarounds.

This issue is so frustrating, because the knee-jerk reaction is to criticize Homebrew on this (see, for example, @ubruhin’s commit message above, “CI: Add one more workaround for buggy homebrew”) — but they’re not actually at fault here.

GitHub caused this, by dumping a bunch of stuff into /usr/local/ outside of Homebrew’s control, but all mixed in with stuff that IS under its control.

I’m sure the Homebrew devs would say that Homebrew isn’t meant to be used that way, and they can’t be responsible for any breakage that it causes… and you know, it’d be absolutely fair for them to take that position.

The only thing Homebrew is doing “wrong” is REFUSING to blow away existing, unexpected files in a directory that they didn’t put there, don’t know where they came from, and can’t make any assumptions about how important they might be or how safe it is to overwrite them.

…Now, having the ability to configure brew into an “always-overwrite” mode for situations like this, if the user accepts full responsibility for any damage that might result… That could possibly be a handy tool for dealing with this situation, given what a mess GitHub has made of their macOS runner images.

But Homebrew is an all-volunteer project that we certainly shouldn’t just go demanding a fix from, and even if they had the spare time to work on this I could certainly sympathize if they were reluctant to do so. Any solution coming from their end would inevitably involve them relaxing safeguards they built into the brew tool, and potentially exposing themselves to backlash should someone immediately aim it at their foot and pull the trigger.

Frustrating. GitHub needs to clue in as to why they’re Doing It Wrong™, with this mess.

After an exhausting debug session I used a big hammer:

https://github.com/mesonbuild/meson/blob/2d0c9ce5f270306610c502859ebd46fd92162fb1/.github/workflows/macos.yml#L72-L77

    # github actions overwrites brew's python. Force it to reassert itself, by running in a separate step.
    - name: unbreak python in github actions
      run: |
        find /usr/local/bin -lname '*/Library/Frameworks/Python.framework/*' -delete
        sudo rm -rf /Library/Frameworks/Python.framework/
        brew install --force python3 && brew unlink python3 && brew unlink python3 && brew link --overwrite python3

@traversaro actully it is a normal behaviour. As you see from the log the installation continues normally with just a warning the links were not overwritten. The problem is brew upgrade exits with non-zero exit code and bash shell runs with set -e option. Thus the step signals the error condition despite brew upgrade is the very last command or not.

The easiest way to make the step to success is to add true after brew upgrade:

      - run: |
          brew update
          brew upgrade || true

So, one point I don’t see covered above:

brew install arguments

$ brew install --help
Usage: brew install [options] formula|cask [...]

Install a formula or cask. Additional options specific to a formula may be
appended to the command.

[....]

  -f, --force                      Install formulae without checking for
                                   previously installed keg-only or non-migrated
                                   versions. When installing casks, overwrite
                                   existing files (binaries and symlinks are
                                   excluded, unless originally from the same
                                   cask).
[...]
      --overwrite                  Delete files that already exist in the prefix
                                   while linking.

Observations:

1. There’s no point in using brew install --force, since it explicitly doesn’t help with this issue. Linking is the one phase that’s NOT forced, even with --force.
2. brew install --overwrite, OTOH, was created specifically to address this issue, and only this issue, when linking installed binaries.

So, rather than manually deleting links (after trying to guess what might need to be deleted), it should be at least as safe to just pass --overwrite to any (and every) brew install call that might need to clobber existing, unexpected symlinks in /usr/local/bin/, and let it take care of things.

I’ve been experimenting with that, and so far it seems to work.

No path for brew upgrade

One caveat: brew upgrade doesn’t support the same --overwrite argument. (Edit: Nor does brew reinstall.) So, if there’s an already-installed package with clobbered symlinks, it appears that doing a brew remove followed by a brew install --overwrite (or, alternatively, a brew install --force --overwrite) is probably the best option.

Looks like issue was upgraded with Could not symlink bin/2to3-3.11

Hello @fredroy, i see you forces the links by removing them manually in fact it is better to be done with

brew link --overwrite python@3.10

FWIW the symlink uninstallation problem is a brew bug IMO.

Hmm, I don’t think so…

What brew is erroring out on is the fact that it has detected, the symlinks in question do not belong to it – although they were supposed to, now they refer to some other software entirely.

As a result, brew defaults to reporting an error because it doesn’t know whether the other software requires those modifications. You can, of course, override this and tell brew to force overwrite the symlinks.

IMHO brew is well behaved here.

I’ve transferred the issue to the actions/setup-python repo as the most relevant place.

@igagis yes, because toolcached python is installed during image creation process, no matter will anybody use it or not it is already a part of the image.

Also observed this here https://github.com/ryancurrah/lima-and-qemu/actions/runs/3745432855/jobs/6359863993 and here https://github.com/mook-as/lima-and-qemu/actions/runs/3743467832/jobs/6356414606.

EDIT:

I was able to get it working by removing the existing symlinks.

rm /usr/local/bin/2to3
rm /usr/local/bin/idle3
rm /usr/local/bin/pydoc3
rm /usr/local/bin/python3
rm /usr/local/bin/python3-config

Hi @traversaro , thank you, we will take a look.

As I said a while back, to me it seems like the safest path is to use brew install --force --overwrite to install over any packages that GitHub has directly installed into /usr/local. (You’ll know which ones they are, because they’re the ones that break brew upgrade.) Even when Homebrew’s version is even with GitHub’s version, that won’t break anything. And if Homebrew’s version is newer, it will let Homebrew successfully update the package.

If GitHub’s version was newer than Homebrew’s, you’ll end up with a version downgrade — but that’s probably not that big a deal, and I don’t expect it’ll happen particularly often, or last for very long if it does.

@davidbarton You’re not wrong there. And I think even the Homebrew people recognized that it wasn’t the best choice, in hindsight, because the Linux default has always been /home/linuxbrew/.linuxbrew, and for Silicon Macs they changed it to /opt/homebrew. But for Intel Macs it’s just too entrenched.

At the same time, as you say, GitHub know they’re operating in a Homebrew-managed environment, and should know what they’re doing is going to cause problems.

(Plus, it’s not like it’s hard to make Homebrew happy. Install into $(brew --cellar)/pkgname/version/, and link from there into $(brew --prefix)/{bin,lib,share,...}. That’s really all it takes.)

@ferdnyc To be fair Homebrew should not have had taken the /usr/local/ into their custody in the first place. It is meant to be used by all users and all programs. But you are right that Github need to fix this mess as they need to play by the rules defined by Homebrew if they wish to use it.

@traversaro indeed, just faced this issue as well.

FWIW, I documented a (manual but systematic) way to workaround the issue in the wiki of our project:

https://github.com/ocaml-sf/learn-ocaml/wiki/(CI)(macOS)-How-to-fix-spurious-brew-link-failures

I had similar issues with Python formulas in Homebrew with my emacs-builds project.

For now, I’ve added a hacky solution that just removes any symlinks in /usr/local/bin which target the system version of Python.

In case it helps anyone:

find /usr/local/bin -type l -ilname '*/Library/Frameworks/Python.framework/*' -delete

The issue is still there. Here is one more build log, if it helps somehow for analysis: https://github.com/cppfw/utki/actions/runs/3881378828/jobs/6740887891#step:4:297

The issue looks very serious and affecting a lot of users, since it is basically any brew install fails. Strange to see it is still not fixed and doesn’t get any attention from maintainers 😦

@igagis the fix should be made in the actions/python-versions repo in first place, it is image - irrelevant.

This is not true. Many people with this issue do not use that action, the conflicting Python symlink targets e.g /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3. See the workflow file from the original issue post: https://github.com/traversaro/github-actions-brew-update-upgrade-check/actions/runs/3742142787/workflow

@dsame

@Flamefire the workflow does not use action/setup-python. It seems the links are created by the macos python package - i see them on the “almost” clean machine with the XCode installed.

Yes this is what I also observed and wrote in https://github.com/actions/setup-python/issues/577

As you see from the log the installation continues normally with just a warning the links were not overwritten. The problem is brew upgrade exits with non-zero exit code

We also see this when we install a package with brew (e.g. ccache) which requires Python and hence is automatically upgraded. Ignoring the exit code is not feasible in that case as it might hint to anything besides this issue.

i see you forces the links by removing them manually in fact it is better to be done with

brew link --overwrite python@3.10

This is also not feasible for us as the exact Python version required for a package to be installed is not known/may change.

Hence our workaround is to remove the symlinks on failure and retry: https://github.com/boostorg/boost-ci/blob/ac5ae7e901e49351d19de278cbf82ce8d2c44216/ci/setup_ccache.sh#L17-L24

@maxkratz

I’m also observing this behavior (just with running brew update and macOS 12): https://github.com/eMoflon/emoflon-ibex-eclipse-build/actions/runs/3764385410/jobs/6398766536

No, I also made this false assumption. The behavior is not triggered by the update (which just downloads meta information similar to apt-get update) but the following installation of a package: brew install p7zip coreutils grep wget curl (or brew upgrade but you don’t use that here)

Got the same issue, and find out that something similar we had previously. Now it looks like:

  # homebrew fails to update python 3.9.1 to 3.9.1.1 due to unlinking failure
  rm /usr/local/bin/2to3 || true
  # homebrew fails to update python from 3.9 to 3.10 due to another unlinking failure
  rm /usr/local/bin/idle3 || true
  rm /usr/local/bin/pydoc3 || true
  rm /usr/local/bin/python3 || true
  rm /usr/local/bin/python3-config || true

I’m observing this as well