cli: Uploading new releases is too slow and mostly times out

Describe the bug

We are trying to upload artefacts created with our build server. We are uploading two files, one ~50Mb and the other ~90Mb. We can upload one or the other, but it takes 20 minutes to do so. If we try to upload both the process fails with http2: Transport: cannot retry err [stream error: stream ID 1; REFUSED_STREAM] after Request.Body was written; define Request.GetBody to avoid this error.

If I run the speedtest CLI tool the upload speed is nearing 100Mb/s, yet while the gh release create command is running it barely reaches 1Mb/s. If I use the website, the upload is very fast.

Our version is gh version 1.7.0 (2021-03-04).

Steps to reproduce the behavior

Type gh release create speed-test build1.tar.gz build2.tar.gz
Wait 20 minutes
Error appears

Expected vs actual behavior

The release should upload relatively quickly and the process should not fail.

Logs

  Starting: /opt/buildagent/temp/agentTmp/custom_script1786572503185520936
18:31:43
  in directory: /opt/buildagent/work/cd80fa15c4f3d102
18:51:31
  Post "https://uploads.github.com/repos/our-team/our-project/releases/release-id/assets?label=&name=our-file.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 1; REFUSED_STREAM] after Request.Body was written; define Request.GetBody to avoid this error
18:51:31
  Process exited with code 1
18:51:31
  Process exited with code 1 (Step: Create GitHub release (Command Line))

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 5
Comments: 33 (13 by maintainers)

Most upvoted comments

I’ve been experiencing similar slowdowns when uploading large (> 163 Mb) files. The performance is just exponentially slowing down. A single 100 Mb file is uploaded in 20ish seconds when a 300 Mb file gets uploaded in almost 3 minutes.

The weird part is that it only getting reproduced from two data centers we have in US. I also tested in one datacenter in Paris and it doesn’t have the same problems.

Note that is seems not CLI specific and I can reproduce the issue with curl. Decided to share my current findings since it might be an issue on the uploads.github.com side and @mislav keeps an eye on this thread.

fkorotkov on Dec 13, 2022

Release upload speeds have been painfully slow for me during the last week.

blipk on Feb 13, 2022

@OldManMeta Max upload speed is the property of the platform, not of the client. This issue tracker is for the GitHub CLI command-line client, which is only able to upload large files as fast as the platform would accept it. If you want to share your feedback about the platform itself, you could post in https://github.com/orgs/community/discussions/categories/repositories or write to GitHub Support. Thanks!

mislav on Apr 10, 2023

@caarlos0 Is the problem the fact that goreleaser uploads eventually fail, or just that they are very slow?

If the uploads fail, does goreleaser retry them?

I can ask relevant teams, but I’m not sure what I can tell them. The reports we’ve received so far are vague and difficult to reproduce and debug. Maybe the best approach would be to create a dedicated thread for this at a place like https://github.com/github/feedback/discussions and have affected parties add as much detail as they can to aid the infrastructure team look into this.

mislav on Apr 5, 2022

Hey, I don’t know if you guys fixed anything, but our upload times have been very good lately. No slow uploads since March 23rd, we’ve had 14 uploads in between and they were all fast. As far as I know nothing changed on our end.

gregopet on Apr 7, 2021