gogs: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Bad Gateway

Like @netlander the workaround of bypassing the Apache proxy worked for me. It was crashing at that step: “POST git-receive-pack (7200216 bytes)”, but this works normally without the proxy.

Might well be related to the 401: with the Apache mod_proxy, I did not receive an authentication request, but I received one without the proxy and had to type user/password for the commit to be accepted.

Apache error log (multiple identical lines):

[Sat Jan 12 09:12:33.710124 2019] [proxy:error] [pid 25787] (32)Broken pipe: [client 37.58.212.212:59962] AH01084: pass request body failed to 127.0.0.1:3000 (127.0.0.1)
[Sat Jan 12 09:12:33.710162 2019] [proxy_http:error] [pid 25787] [client 37.58.212.212:59962] AH01097: pass request body failed to 127.0.0.1:3000 (127.0.0.1) from 37.58.212.212 ()

Gogs log/hooks/post-receive.log (9:20 is without the proxy, nothing visible at 9:12 with the proxy):

2019/01/12 09:20:41 [ERROR] [...ogs/models/action.go:493 CommitRepoAction()] UpdateIssuesCommit: invalid issue reference [ref: 09/01/2019]

Other log/hooks/* files are touched but empty

Portion of the log/gogs.log which appears relevant, without the proxy:

2019/01/12 09:20:41 [TRACE] HTTPGit - Authenticated user: fgm
2019/01/12 09:20:41 [TRACE] Session ID: d8376360d674a0ea
2019/01/12 09:20:41 [TRACE] CSRF Token: FKyzeZELo1j4eoGE-TKp3yDmyc06MTU0NzI4NDg0MTU0MjMxMjM2MA==
2019/01/12 09:20:41 [TRACE] Detected encoding: UTF-8 (fast)
2019/01/12 09:20:41 [TRACE] Template: repo/home
2019/01/12 09:20:41 [TRACE] Session ID: 9e0fc1184eb4970b
2019/01/12 09:20:41 [TRACE] CSRF Token: M0fhCH0kbQAhgfPKdmaXSga1DPk6MTU0NzI4NDg0MTkxMjIwMDAwOA==
2019/01/12 09:20:41 [TRACE] TriggerTask 'assets__prague2013_blocks/blocks-1-8' by 'fgm'
2019/01/12 09:20:41 [TRACE] DeliverHooks [repo_id: 112]
2019/01/12 09:20:41 [TRACE] AddTestPullRequestTask [head_repo_id: 112, head_branch: blocks-1-8]: finding pull requests
2019/01/12 09:20:41 [TRACE] AddTestPullRequestTask [base_repo_id: 112, base_branch: blocks-1-8]: finding pull requests

More interestingly, updating my HTTPS Vhost to look like this, as suggested on https://httpd.apache.org/docs/2.4/en/mod/mod_proxy.html#envsettings , fixed the issue for me. It causes the browser to ask for credentials (triggering keychain access in macOS), and succeeds:

  ProxyPreserveHost On
  ProxyRequests off
  ProxyPass / http://127.0.0.1:3000/
  ProxyPassReverse / http://127.0.0.1:3000/
  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1

This might suggest a problem with HTTP/1.1 keepalive handling in Gogs or its dependencies.

I had this error with Gitlab behind an Nginx proxy with pushing large files.

$ git push origin master
warning: redirecting to http://xxxx.xxxx.xxx/xxxx/xxxx.git/
Uploading LFS objects: 100% (2/2), 408 MB | 0 B/s, done                                                                                                                                                                                                 
Enumerating objects: 8, done.
Counting objects: 100% (8/8), done.
Delta compression using up to 6 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 386.08 MiB | 8.74 MiB/s, done.
Total 8 (delta 0), reused 0 (delta 0)
error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

This was my solution for nginx based on @fgm and the keepalive clue: (SSL terminated at a higher layer, this is for the internal LB in front of docker containers) Narrowed down the issue to proxy_http_version, keepalive and removing the connection header.

http {
...
    upstream gitlab {
        keepalive 100;
        server 00.000.0.0:8080;
    }

server {
        listen 80;
        server_name gitlab.entouch.jp;

        location / {
            add_header Strict-Transport-Security "max-age=15768000;";
            keepalive_timeout 120s;

            proxy_read_timeout 3600s;
            proxy_http_version 1.1;

            # Remove Connection header, client might send "close" to close a keepalive connection
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Host $host:$server_port;
            proxy_set_header X-Forwarded-Server $host;

            proxy_pass http://gitlab/;
        }
    }
...
}

Something else that can help to narrow down the cause: GIT_CURL_VERBOSE=1 GIT_TRACE=1 git push origin master

Ran also into this issue today. We have several repos, and only one cannot be pushed anymore. Using apache2.2, gogs 0.11.34 and reverse proxy. SetEnv proxy-sendcl 1 didn’t help. Setting the git http.postBuffer to 524288000 didn’t help.

The workaround that helped was to bypass the proxy. So I changed gogs to listen on the lan ip and change the remote url to connect to port 3000 directly. Then I could push, revert above changes, and newly pushed commits are working again.

So this seems to be a strange combination of things. The problematic push had 5 commits, maybe it’s a size problem?

@Unknwon installed Caddy, no problems. I am assuming it was an Apache, a mod_proxy or a mod_proxy_http problem. Weird it did not presented itself with other applications, but gogs. Shrugs.

Another tidbit. I installed Gitea (the bastard fork 😂) and it works fine, so it seems the issue is related to Gogs.