gitea: Pushing large amount of data via git-lfs results in HTTP 413 after some time

Gitea version (or commit ref): 1.2.3
Git version: 2.11.0 on server, 2.15.0 on client. git-lfs version 2.3.4 on client and server
Operating system: Debian Stable
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant

The try-instance seems to use git-lfs (if enabled) via https and does not accept my login credentials

Log gist:

Description

I have problems pushing a very large repository (1.8GB) with git lfs to my gitea instance. git pushes the first 80MB, after that, the counter of already uploaded data jumps between weird values (e.g. it increases, decreases, increases again, and so on), and after some time, i get lots of LFS client errors, which are HTTP 413 status codes reported back from gitea. The last few lines say that there is an authorization error for info/lfs/objects/batch and the push process stops. This is reproducible for several push attempts. I tried setting the log level to Debug, but the log file stays quite empty.

So i’m asking for help for how to set up Debug logging (i cannot find a list of proper log level options), and i want to notice for this issue in general.

I have configured a Bitbucket Repository as a second remote, and it accepts the big push without problems, so i think it is a server related issue.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 32 (6 by maintainers)

Most upvoted comments

This maybe a reverse proxy upload size setting problem?

lunny on Nov 17, 2017

Damn, it turned to be just a strange ingress issue. I drilled down the Nginx ingress controller and for some reason, it was ignoring any new annotations I was adding to the ingress spec. Deleting the ingress and bringing it back solved the issue! Thanks @theAkito and @sapk for your quick feedback!

UPDATE Still, it doesn’t work over https however now the problem is ~~identified as ingress problem~~.

nikitaeverywhere on Apr 29, 2020

@bthulu Perhaps you should open a new issue. It is working for most people, as far as I have seen (including myself). Maybe you have a different root issue.

theAkito on May 19, 2020

I am experiencing this same bug. Has this issue been fixed? I am using git-lfs/2.11.0 (GitHub; linux amd64; go 1.13.4)

Update. I was able to fix this issue with git config http.version HTTP/1.1

kaleko on May 15, 2020

@ZitRos That is good to know, especially for future readers of this issue.

theAkito on Apr 29, 2020

UPDATE2: YAY! Turns out it’s git-lfs problem, which doesn’t handle 308 redirects properly which results in “file already closed” errors. The latest git-lfs binary (which is not yet released!) works flawlessly!

nikitaeverywhere on Apr 29, 2020

UPDATE 3 pushing via HTTP is not successful either, however, now it seems not to push because of the size of the objects (~>48mb). @theAkito thanks! I keep trying 😄

nikitaeverywhere on Apr 29, 2020

@sapk thanks for suggestions! I tried everything 😃

metadata:
  name: gitea-http
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 5000m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: 3600
    nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
    nginx.ingress.kubernetes.io/proxy-read-timeout: 3600
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: 3600
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

Actually I don’t think the timeout is a problem, as underneath git CLI issues a lot of individual uploads which take ~15s at max.

I even tried the direct connection to k8s service using NodePort (without ingress), no luck there as well, but now it fails with 413 after ~1min 30s instead of 1 min 😃 Unfortunately, git CLI seems to start over all the time, which doesn’t allow me to upload all the objects even with this strange errors.

Gitea pod logs say nothing about a bunch of 413’s I see after executing git push but just a number of successful uploads:

[Macaron] 2020-04-29 18:56:36: Started POST /user/name.git/info/lfs/objects/batch for 10.1.27.1
[Macaron] 2020-04-29 18:56:37: Completed POST /user/name.git/info/lfs/objects/batch 200 OK in 967.411097ms

To add even more details, I am uploading ~1.5 Gb of data which are large jpegs of 5-100mb.

UPDATE by looking at git CLI debug trace more closely, I figured out that it certainly fails to upload some files. (not to mention it always try to http and then gets 308 to https)

git lfs push zitroserver --object-id 55ac260c6a81ab0ddd71be1784f9c13b5aff2ccfe99510b0e24a461e3043627a

Uploading LFS objects:   0% (0/1), 688 KB | 248 KB/s, done.
LFS: Put https://git.svc.nikita.tk/ZitRos/nikita-tk.git/info/lfs/objects/55ac260c6a81ab0ddd71be1784f9c13b5aff2ccfe99510b0e24a461e3043627a: read C:\ZitRo\Projects\Personal\nikita-tk\.git\lfs\objects\55\ac\55ac260c6a81ab0ddd71be1784f9c13b5aff2ccfe99510b0e24a461e3043627a: file already closed

55ac260c6a81ab0ddd71be1784f9c13b5aff2ccfe99510b0e24a461e3043627a is 178kb file!

UPDATE 2 wasting a bit more time I figured out that this issue is related to http->https redirect. Even the direct connection (Gitea server) returns http URL (external host as set up in settings), and then gets 308 to https from ingress. Uploading via https for some reason fails, and only for some files…

nikitaeverywhere on Apr 29, 2020

60s seems to be the default timeout of nginx. You can try to adjust them (like proxy_read_timeout). http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream_timeout

sapk on Apr 29, 2020

I’m having the same issue: after ~60sec of uploading to LFS it fails.

LFS: Client error: http://host.com/user/repo/info/lfs/objects/9d49bf70e8f00e0a815b07ffc8fbce4aa521e6e303f19c2d173fa5d8e1e518c8 from HTTP 413

I am using Kubernetes with nginx ingress (installed from Helm), and playing with nginx.ingress.kubernetes.io/proxy-body-size didn’t help.

nikitaeverywhere on Apr 28, 2020

I added this in nginx configure file

http {
     # ..........
     # at the END of this segment!
     client_max_body_size 3000m;
}

Fixed it. Thanks, @liu-kan !

theAkito on Oct 29, 2019