gitea: Migration of repositories with tags to organisation fails
- Gitea version (or commit ref): 1.9.4
- Git version: 1.8.3.1
- Operating system: CentOS 7
- Database (use
[x]):- PostgreSQL
- MariaDB
- MSSQL
- SQLite
- Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant
- Log gist:
Description
We are currently migrating all of our 230 repositories to Gitea. Here’s how we do it:
- git clone --mirror https://old-git.example.org/repo.git
- git push --mirror https://gitea.example.org/ORG/repo.git
To keep it simple our Gitea instance has just one organisation with around 110 “Owner” team members. When we migrate repositories with more than 50 tags to that organisation, the git client pushes everything without an error, but Gitea fails when creating releases for every tag with the following message:
2019/10/16 11:12:42 models/webhook.go:909:DeliverHooks() [E] Get repository [235] hook tasks: dial tcp 172.xxx.xxx.xxx:3306: connect: cannot assign requested address
In the Gitea logfile I can see that Gitea calls repo_permission.go for every tag and every user of that organisation, which results in a lot of calls and presumably database queries:
2019/10/16 11:12:39 models/update.go:80:PushUpdate() [T] TriggerTask 'TAGNAME' by gituser
2019/10/16 11:12:39 models/pull.go:1088:AddTestPullRequestTask() [T] AddTestPullRequestTask [head_repo_id: xxx, head_branch: xxx]: finding pull requests
2019/10/16 11:12:39 modules/auth/auth.go:97:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid
2019/10/16 11:12:39 ...s/context/context.go:328:func1() [D] Session ID: xxx
2019/10/16 11:12:39 ...s/context/context.go:329:func1() [D] CSRF Token: xxx
2019/10/16 11:12:39 models/pull.go:1128:AddTestPullRequestTask() [T] AddTestPullRequestTask [base_repo_id: xxx, base_branch: xxx]: finding pull requests
2019/10/16 11:12:40 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824725727744:USER1 in 824725727824:ORG/reponame
2019/10/16 11:12:40 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824729407712:USER2 in 824729407808:ORG/reponame
2019/10/16 11:12:40 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824729900800:USER3 in 824729900880:ORG/reponame
[...]
When migrating that same repository to a “standalone” user account, everything works as expected. All releases get created. In the Gitea logfile I can see that gitea doesn’t call repo_permission.go, in this case:
2019/10/16 12:18:27 models/update.go:80:PushUpdate() [T] TriggerTask 'TAG1' by gituser
2019/10/16 12:18:27 models/pull.go:1088:AddTestPullRequestTask() [T] AddTestPullRequestTask [head_repo_id: xxx, head_branch: xxx]: finding pull requests
2019/10/16 12:18:27 modules/auth/auth.go:97:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid
2019/10/16 12:18:27 ...s/context/context.go:328:func1() [D] Session ID: xxx
2019/10/16 12:18:27 ...s/context/context.go:329:func1() [D] CSRF Token: xxx
2019/10/16 12:18:27 models/pull.go:1128:AddTestPullRequestTask() [T] AddTestPullRequestTask [base_repo_id: xxx, base_branch: xxx]: finding pull requests
2019/10/16 12:18:28 models/webhook.go:898:DeliverHooks() [T] DeliverHooks [repo_id: xxx]
2019/10/16 12:18:28 models/update.go:80:PushUpdate() [T] TriggerTask 'TAG2' by gituser
[...]
We also tried to migrate a very large repository with around 15.000 commits and 7000 tags. Although we get an error from the git client, everything (tags, commits) gets transmitted and all releases get created by gitea when migrating to a user repository.
user@linux:~/repo.git$ git push --mirror https://gitea.example.org/ORG/repo.git
Counting objects: 409616, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (108168/108168), done.
Writing objects: 100% (409616/409616), 1.14 GiB | 25.52 MiB/s, done.
Total 409616 (delta 201291), reused 409611 (delta 201286)
error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer
Everything up-to-date
user@linux:~/repo.git$ git push --mirror https://gitea.example.org/ORG/repo.git
Everything up-to-date
Long story short, when we migrate repositories with more, or way more than 50 tags to a user account, everything works. When we migrate them to an organizazion with a lot of members, Gitea gets stuck when creating releases from tags.
Workaround
As a workaround we first migrate to a user account and then transmit the ownership of the repository to the organisation.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 29 (20 by maintainers)
I just had the chance to test with a build from @zeripath PR and the following db settings:
With this build and settings I can see a proper database pooling. During migration I have two or three tcp sockets in
ESTABLISHEDstate, which get used randomly. After five minutes the sockets go in TIME_WAIT and then get removed, as expected. With this build the migration of huge repos with a lot of tags works well, even though it’s pretty slow. But all releases get created properly. I think #8602 should improve the speed issue. Hope to see both fixes/improvements in a official release soon.@guillep2k I will try that on monday when I am back at the office and get back to you.
Without looking at the code I would guess it’s actually “random” - the most obvious implementation is a simple spin lock with a wait until you actually get the real lock. I would bet there is no formal queue - too expensive - so we’re into OS level queuing algorithms, suggesting a likely bias towards LIFO.
@zeripath Mmm… I was about to write a long explanation of how MaxOpenConns should not affect the number of closed connections but now I think I see your point. The only way to avoid the system from creating time_wait entries is to keep it from closing them as much as possible, so MaxIdleConns should be equal to MaxOpenConns in this type of application where many users can be doing operations at the same time.
Again, your PR seems on point. What I wonder is: what’s the strategy in the database driver for scheduling the connection requests when they are all busy? FIFO? Are there others available?
Is easier to read.