brew: custom download strategy no longer works after commit fbf474a3fd107b636eb397b0aa247f7e03f2dc72

`brew doctor` output

Your system is ready to brew.

Verification

My “brew doctor output” above says Your system is ready to brew. and am still able to reproduce my issue.
I ran brew update twice and am still able to reproduce my issue.
This issue’s title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

`brew config` output

HOMEBREW_VERSION: 4.0.11-95-gd15f571
ORIGIN: https://github.com/Homebrew/brew
HEAD: d15f571eb6a994e7ca689721a54c5c6bff4219a0
Last commit: 28 hours ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 29bc842d5cdffde2917bce61a1d4eb742cf6d987
Core tap last commit: 7 hours ago
Core tap branch: master
Core tap JSON: 06 Apr 17:27 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 16
HOMEBREW_NO_INSTALL_FROM_API: set
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 16-core 64-bit kabylake
Clang: 14.0.3 build 1403
Git: 2.40.0 => /usr/local/bin/git
Curl: 7.87.0 => /usr/bin/curl
macOS: 13.3-x86_64
CLT: 14.3.0.0.1.1679647830
Xcode: 14.3

What were you trying to do (and why)?

I try to “brew install” a private repo, with a custom download strategy configured.

The download strategy is taken from https://dev.to/jhot/homebrew-and-private-github-repositories-1dfh.

The strategy used is GitHubPrivateRepositoryReleaseDownloadStrategy.

If I roll back the commit, referenced by fbf474a3fd107b636eb397b0aa247f7e03f2dc72, it works.

What happened (include all command output)?

❯ brew reinstall handelsblattgroup/tap/kundetools                                                                                                                                                                                                          
==> Fetching handelsblattgroup/tap/kundetools
==> Downloading https://github.com/handelsblattgroup/kunde2go-tools/releases/download/v3.0.0/kunde2go-tools_Darwin_x86_64.tar.gz
Error: Failed to download resource "kundetools"
Failure while executing; `/usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.0.11-95-gd15f571\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 13.3\)\ curl/7.87.0 --header Accept-Language:\ en --retry 3 --fail --location --silent --head --request GET https://github.com/handelsblattgroup/kunde2go-tools/releases/download/v3.0.0/kunde2go-tools_Darwin_x86_64.tar.gz` exited with 22. Here's the output:
curl: (22) The requested URL returned error: 404
HTTP/2 404
server: GitHub.com
date: Fri, 07 Apr 2023 14:17:13 GMT
content-type: text/plain; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
cache-control: no-cache
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'; base-uri 'self'; connect-src 'self'; form-action 'self'; img-src 'self' data:; script-src 'self'; style-src 'unsafe-inline'
content-length: 9
x-github-request-id: 9484:FD42:57B0878:591A642:643025E8

What did you expect to happen?

❯ brew reinstall handelsblattgroup/tap/kundetools
==> Fetching handelsblattgroup/tap/kundetools ==> Downloading https://github.com/handelsblattgroup/kunde2go-tools/releases/download/v3.0.0/kunde2go-tools_Darwin_x86_64.tar.gz Already downloaded: /Users/xxx/Library/Caches/Homebrew/downloads/7e981c9e344d9fd5524ce1dc953191b2beceb8efba5c8283a53c1a0e4fbc4856–kunde2go-tools_Darwin_x86_64.tar.gz ==> Reinstalling handelsblattgroup/tap/kundetools ==> Caveats zsh completions have been installed to: /usr/local/share/zsh/site-functions ==> Summary 🍺 /usr/local/Cellar/kundetools/3.0.0: 6 files, 13.6MB, built in 4 seconds ==> Running brew cleanup kundetools… Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP. Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).`

Step-by-step reproduction instructions (by running `brew` commands)

brew up
brew reinstall handelsblattgroup/tap/kundetools

About this issue

Original URL
State: closed
Created a year ago
Reactions: 22
Comments: 21 (9 by maintainers)

Commits related to this issue

Update our custom download strategy that was broken due to brew changes Brew changed how CurlDownloadStrategy works in a recent commit which broke our custom download strategy. Ref. https://github... — committed to superbet-group/homebrew-betting.platform by ifutivic a year ago
fix: make downloads work with homebrew 4 see https://github.com/Homebrew/brew/issues/15169#top — committed to meetcleo/homebrew-cleo by flood4life a year ago

Most upvoted comments

I’m having this same issue. I am also using the private release download strategy very similar to the one linked above. I was able to fix this by adding this override to my GitHubPrivateRepositoryReleaseDownloadStrategy

  def resolve_url_basename_time_file_size(url, timeout: nil)
    [download_url, "", Time.now, 0, false]
  end

Any advice on how to fix this on my end in a more robust way?

+30

Brunope on Apr 7, 2023

Okay, so for private GitHub releases, it should be possible to do this without any custom download strategy by specifying the headers in the formula and using the API URL directly, e.g.

  url "https://api.github.com/repos/Homebrew/homebrew-portable-ruby/releases/assets/103219629",
      headers: [
        "Accept: application/octet-stream",
        "Authorization: bearer #{ENV["HOMEBREW_GITHUB_API_TOKEN"]}"
      ]

Of course you’ll likely have to specify version manually in this case. Alternatively, you’ll need to complain to GitHub and demand they support private release downloads with the standard URL and a Authorization header to make this easier.

+10

reitermarkus on Apr 12, 2023

I think I have already said this a few years ago when this broke in a similar way:

We need to deprecate this way of allowing custom download strategies and replace it with a simpler API with less surface area.

It’s way too brittle in the current state since only the fetch method should be overriden but the fetch in CurlDownloadStrategy is way too complex, and technically you also need to override resolved_basename in the case of GitHub releases so it doesn’t 404 on private repos.

reitermarkus on Apr 7, 2023

@jrouly We don’t support installing from private repositories and wouldn’t use it ourselves so: no, we’re not good candidates for maintainers for this code.

MikeMcQuaid on Apr 7, 2023

We ran into this as well. The change suggested by @Brunope resolved the issue for us as well.

It sounds like private repository support won’t come in homebrew core. @MikeMcQuaid do you have any guidance on how the community better manages this in the future in order to avoid these codebase conflicts? Homebrew is a great package manager for opensource packages, which is why other organizations (perhaps startups like mine) rely on an existing and known tool for distributing internal packages. The way it sounds like we’re all accomplishing this is adding per-tap custom download strategies that each org needs to update every time there is a download strategy base class change.

drn on Apr 11, 2023

We were able to workaround this by overriding the resolve_url_basename_time_file_size in our strategy:

  def resolve_url_basename_time_file_size(url, timeout: nil)
    url = download_url
    super
  end

We specifically are using this gist for our private strategy.

mike-carey on Apr 11, 2023

I stumbled upon the same issue, and Inheriting from AbstractFileDownloadStrategy fixed my problem.

Note that AbstractFileDownloadStrategy is a private API, so we really should find a solution that doesn’t require inheriting from it.

Either that, or make the API public and then add unit tests to make sure we don’t break it again.

carlocab on Apr 11, 2023

Seeing similar recent breakage with our internal Artifactory downloads

dtseiler on Apr 11, 2023

Is it accurate to say that the stance of the Homebrew project is:

Homebrew does not intend to ship and maintain a built-in download strategy that enables downloads from private repos; and
If folks do want to write their own custom download strategies to enable this, they should inherit directly from AbstractFileDownloadStrategy, rather than CurlDownloadStrategy (since CurlDownloadStrategy doesn’t offer an API-stable mechanism for modifying all outbound HTTP requests)?

benweint on Apr 10, 2023