setup-python: brew update followed by brew upgrade fails for Python's 2to3 conflict (December 2022)
Description
If one has on a job:
brew update
brew upgrade
the job fails with error:
==> Upgrading python@3.10
3.10.8 -> 3.10.9
==> Pouring python@3.10--3.10.9.big_sur.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
rm '/usr/local/bin/2to3'
To force the link and overwrite all conflicting files:
brew link --overwrite python@3.10
To list all files that would be deleted:
brew link --overwrite --dry-run python@3.10
Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3-config
This is similar to old issues:
- https://github.com/actions/runner-images/issues/2322
- https://github.com/actions/runner-images/issues/4020
Platforms affected
- Azure DevOps
- GitHub Actions - Standard Runners
- GitHub Actions - Larger Runners
Runner images affected
- Ubuntu 18.04
- Ubuntu 20.04
- Ubuntu 22.04
- macOS 11
- macOS 12
- Windows Server 2019
- Windows Server 2022
Image version and build link
Image Version: 20221215.1 Link: https://github.com/traversaro/github-actions-brew-update-upgrade-check/actions/runs/3742142787
Is it regression?
20221211.1
Expected behavior
The brew upgrade
should exit fine.
Actual behavior
The brew upgrade
fails with the error message in the top of the issue.
Repro steps
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 11
- Comments: 49 (11 by maintainers)
Commits related to this issue
- feat(release): update macos version used by github action MacOS Catalina is no longer supported by Apple or GitHub https://github.blog/changelog/2022-07-20-github-actions-the-macos-10-15-actions-runn... — committed to ryancurrah/lima-and-qemu by ryancurrah 2 years ago
- Ignore errors of `brew update` in setup_ccache It may fail creating some symlinks which we don't care about. See e.g. https://github.com/actions/runner-images/issues/6817 — committed to boostorg/boost-ci by Flamefire 2 years ago
- Resolve dependencies intall break on macOS see: https://github.com/actions/runner-images/issues/6817 — committed to shun2wang/jasp-actions by shun2wang 2 years ago
- Resolve dependencies intall break on macOS see: https://github.com/actions/runner-images/issues/6817 — committed to shun2wang/jasp-actions by shun2wang 2 years ago
- Resolve dependencies intall break on macOS see: https://github.com/actions/runner-images/issues/6817 — committed to shun2wang/jasp-actions by shun2wang 2 years ago
- Resolve dependencies intall break on macOS see: https://github.com/actions/runner-images/issues/6817 Signed-off-by: Shun Wang <shuonwang@gmail.com> — committed to shun2wang/jasp-actions by shun2wang 2 years ago
- GitHub Actions: use python 3.10 instead of 3.11 on MacOS See https://github.com/actions/runner-images/issues/6817 — committed to masatake/ctags by masatake 2 years ago
- GitHub Actions: use python 3.10 instead of 3.11 on MacOS See https://github.com/actions/runner-images/issues/6817 Suggested-by: leleliu008 <leleliu008@gmail.com> — committed to masatake/ctags by masatake 2 years ago
- GitHub Actions: use python 3.10 instead of 3.11 on MacOS See https://github.com/actions/runner-images/issues/6817 Suggested-by: leleliu008 <leleliu008@gmail.com> — committed to universal-ctags/ctags by masatake 2 years ago
- Remove brew update from macOS runner workaround for https://github.com/actions/runner-images/issues/6817 — committed to awawa-dev/HyperHDR by awawa-dev 2 years ago
- Try to fix ci pipline python error - Follow this issue: https://github.com/actions/setup-python/issues/577 — committed to sbearben/dotfiles by sbearben 2 years ago
- CI & CD: fix for https://github.com/actions/setup-python/issues/577 — committed to bigcat88/pillow_heif by bigcat88 2 years ago
- Hopefully MACOS CI will get fixed here https://github.com/actions/setup-python/issues/577 — committed to cbritopacheco/rodin by cbritopacheco 2 years ago
- Try and fix macos build see https://github.com/actions/setup-python/issues/577 — committed to libsidplayfp/libsidplayfp by drfiemost 2 years ago
- CI: Add one more workaround for buggy homebrew See https://github.com/actions/setup-python/issues/577. — committed to LibrePCB/LibrePCB by ubruhin a year ago
- CI: Add one more workaround for buggy homebrew See https://github.com/actions/setup-python/issues/577. — committed to LibrePCB/LibrePCB by ubruhin a year ago
- [setup] Hotfix for brew failing to upgrade python@3.11 Continuation of #250. Relates: https://github.com/actions/setup-python/issues/577 — committed to RobotLocomotion/drake-external-examples by svenevs a year ago
- [setup] Hotfix for brew failing to upgrade python@3.11 Continuation of #250. Relates: https://github.com/actions/setup-python/issues/577 — committed to RobotLocomotion/drake-external-examples by svenevs a year ago
- [setup] Hotfix for brew failing to upgrade python@3.11 Continuation of #250. Relates: https://github.com/actions/setup-python/issues/577 — committed to RobotLocomotion/drake-external-examples by svenevs a year ago
- [setup] Hotfix for brew failing to upgrade python@3.11 (#252) Continuation of #250. Relates: https://github.com/actions/setup-python/issues/577 — committed to RobotLocomotion/drake-external-examples by svenevs a year ago
Any chance to have it fixed in the image? All was working before, I wouldn’t want to use any workarounds.
This issue is so frustrating, because the knee-jerk reaction is to criticize Homebrew on this (see, for example, @ubruhin’s commit message above, “CI: Add one more workaround for buggy homebrew”) — but they’re not actually at fault here.
GitHub caused this, by dumping a bunch of stuff into
/usr/local/
outside of Homebrew’s control, but all mixed in with stuff that IS under its control.I’m sure the Homebrew devs would say that Homebrew isn’t meant to be used that way, and they can’t be responsible for any breakage that it causes… and you know, it’d be absolutely fair for them to take that position.
The only thing Homebrew is doing “wrong” is REFUSING to blow away existing, unexpected files in a directory that they didn’t put there, don’t know where they came from, and can’t make any assumptions about how important they might be or how safe it is to overwrite them.
…Now, having the ability to configure
brew
into an “always-overwrite” mode for situations like this, if the user accepts full responsibility for any damage that might result… That could possibly be a handy tool for dealing with this situation, given what a mess GitHub has made of their macOS runner images.But Homebrew is an all-volunteer project that we certainly shouldn’t just go demanding a fix from, and even if they had the spare time to work on this I could certainly sympathize if they were reluctant to do so. Any solution coming from their end would inevitably involve them relaxing safeguards they built into the
brew
tool, and potentially exposing themselves to backlash should someone immediately aim it at their foot and pull the trigger.Frustrating. GitHub needs to clue in as to why they’re Doing It Wrong™, with this mess.
After an exhausting debug session I used a big hammer:
https://github.com/mesonbuild/meson/blob/2d0c9ce5f270306610c502859ebd46fd92162fb1/.github/workflows/macos.yml#L72-L77
@traversaro actully it is a normal behaviour. As you see from the log the installation continues normally with just a warning the links were not overwritten. The problem is
brew upgrade
exits with non-zero exit code andbash
shell runs withset -e
option. Thus the step signals the error condition despitebrew upgrade
is the very last command or not.The easiest way to make the step to success is to add
true
afterbrew upgrade
:So, one point I don’t see covered above:
brew install
argumentsObservations:
brew install --force
, since it explicitly doesn’t help with this issue. Linking is the one phase that’s NOT forced, even with--force
.brew install --overwrite
, OTOH, was created specifically to address this issue, and only this issue, when linking installed binaries.So, rather than manually deleting links (after trying to guess what might need to be deleted), it should be at least as safe to just pass
--overwrite
to any (and every)brew install
call that might need to clobber existing, unexpected symlinks in/usr/local/bin/
, and let it take care of things.I’ve been experimenting with that, and so far it seems to work.
No path for
brew upgrade
One caveat:
brew upgrade
doesn’t support the same--overwrite
argument. (Edit: Nor doesbrew reinstall
.) So, if there’s an already-installed package with clobbered symlinks, it appears that doing abrew remove
followed by abrew install --overwrite
(or, alternatively, abrew install --force --overwrite
) is probably the best option.Looks like issue was upgraded with
Could not symlink bin/2to3-3.11
Hello @fredroy, i see you forces the links by removing them manually in fact it is better to be done with
brew link --overwrite python@3.10
Hmm, I don’t think so…
What brew is erroring out on is the fact that it has detected, the symlinks in question do not belong to it – although they were supposed to, now they refer to some other software entirely.
As a result, brew defaults to reporting an error because it doesn’t know whether the other software requires those modifications. You can, of course, override this and tell brew to force overwrite the symlinks.
IMHO brew is well behaved here.
I’ve transferred the issue to the actions/setup-python repo as the most relevant place.
@igagis yes, because toolcached python is installed during image creation process, no matter will anybody use it or not it is already a part of the image.
Also observed this here https://github.com/ryancurrah/lima-and-qemu/actions/runs/3745432855/jobs/6359863993 and here https://github.com/mook-as/lima-and-qemu/actions/runs/3743467832/jobs/6356414606.
EDIT:
I was able to get it working by removing the existing symlinks.
Hi @traversaro , thank you, we will take a look.
As I said a while back, to me it seems like the safest path is to use
brew install --force --overwrite
to install over any packages that GitHub has directly installed into/usr/local
. (You’ll know which ones they are, because they’re the ones that breakbrew upgrade
.) Even when Homebrew’s version is even with GitHub’s version, that won’t break anything. And if Homebrew’s version is newer, it will let Homebrew successfully update the package.If GitHub’s version was newer than Homebrew’s, you’ll end up with a version downgrade — but that’s probably not that big a deal, and I don’t expect it’ll happen particularly often, or last for very long if it does.
@davidbarton You’re not wrong there. And I think even the Homebrew people recognized that it wasn’t the best choice, in hindsight, because the Linux default has always been
/home/linuxbrew/.linuxbrew
, and for Silicon Macs they changed it to/opt/homebrew
. But for Intel Macs it’s just too entrenched.At the same time, as you say, GitHub know they’re operating in a Homebrew-managed environment, and should know what they’re doing is going to cause problems.
(Plus, it’s not like it’s hard to make Homebrew happy. Install into
$(brew --cellar)/pkgname/version/
, and link from there into$(brew --prefix)/{bin,lib,share,...}
. That’s really all it takes.)@ferdnyc To be fair Homebrew should not have had taken the
/usr/local/
into their custody in the first place. It is meant to be used by all users and all programs. But you are right that Github need to fix this mess as they need to play by the rules defined by Homebrew if they wish to use it.@traversaro indeed, just faced this issue as well.
FWIW, I documented a (manual but systematic) way to workaround the issue in the wiki of our project:
https://github.com/ocaml-sf/learn-ocaml/wiki/(CI)(macOS)-How-to-fix-spurious-brew-link-failures
I had similar issues with Python formulas in Homebrew with my emacs-builds project.
For now, I’ve added a hacky solution that just removes any symlinks in
/usr/local/bin
which target the system version of Python.In case it helps anyone:
The issue is still there. Here is one more build log, if it helps somehow for analysis: https://github.com/cppfw/utki/actions/runs/3881378828/jobs/6740887891#step:4:297
The issue looks very serious and affecting a lot of users, since it is basically any
brew install
fails. Strange to see it is still not fixed and doesn’t get any attention from maintainers 😦This is not true. Many people with this issue do not use that action, the conflicting Python symlink targets e.g
/Library/Frameworks/Python.framework/Versions/3.11/bin/2to3
. See the workflow file from the original issue post: https://github.com/traversaro/github-actions-brew-update-upgrade-check/actions/runs/3742142787/workflow@dsame
Yes this is what I also observed and wrote in https://github.com/actions/setup-python/issues/577
We also see this when we install a package with brew (e.g. ccache) which requires Python and hence is automatically upgraded. Ignoring the exit code is not feasible in that case as it might hint to anything besides this issue.
This is also not feasible for us as the exact Python version required for a package to be installed is not known/may change.
Hence our workaround is to remove the symlinks on failure and retry: https://github.com/boostorg/boost-ci/blob/ac5ae7e901e49351d19de278cbf82ce8d2c44216/ci/setup_ccache.sh#L17-L24
@maxkratz
No, I also made this false assumption. The behavior is not triggered by the update (which just downloads meta information similar to
apt-get update
) but the following installation of a package:brew install p7zip coreutils grep wget curl
(orbrew upgrade
but you don’t use that here)Got the same issue, and find out that something similar we had previously. Now it looks like:
I’m observing this as well