gitea: Webhooks randomly fail with 408 timeout

Gitea version (or commit ref): 1.6.0 (latest stable release
Git version: git version 2.17.1 (on the server)
Operating system: Ubuntu Server 18.04
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- Not relevant
Log gist: ?

Description

I’ve setup a webhook between my server and a server at a remote location, I’m getting random failures. Some webhooks go through fine, and others die for some strange reason.

Setup:

Git Server:
- Gitea behind nginx
Remote server:
- webhook behind nginx

Gitea claims the following in the webhook log:

Delivery: Post https://ci.starbeamrainbowlabs.com/hooks/laminar-config-check: read tcp 5.196.73.75:54504->x.y.z.w:443: i/o timeout

…but the remote server’s nginx claims that it’s Gitea’s fault for not sending a complete request in time:

5.196.73.75 - - [04/Dec/2018:20:48:05 +0000] "POST /hooks/laminar-config-check HTTP/1.1" 408 0 "-" "GiteaServer"

The remote server has a good-quality fibre line. The git server has a stable 100mb/sec connection from KimSufi / OVH. Why the timeout?

Annoyingly, it always works if I hit “test delivery” - it’s only genuine push events that are causing problems.

Screenshots

selection_094

Edit: Nope, it doesn’t always work with the “test delivery” button. I’ve just got this:

selection_095

…clicking the “test delivery” button again causes the next one to succeed though.

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 5
Comments: 25 (3 by maintainers)

Most upvoted comments

same problem…

I have set DELIVER_TIMEOUT=30 and still timeout.

I execute curl <webhook URL> command inside my gitea container, it’s very fast and no problem.

I test webhook within another repository, it’s worked. This repository’s webhook URL is same with previous repository’s webhook URL, all they are my Drone CI adress. How can it time out?

ok. I have solved this problem. my gitea repository’s webhook is Drone URL. and the point is that CI repository - setting - Main - Configuration specified yaml file named .drone.yml, and my git repository’s file named drone.yml!! that’s worker after modified file name… 😃

However, this problem should not caused timeout error, network has no problem!

Veitor on Mar 3, 2020

Have the same problem It might be a network problem. Does Gitea have a “retry when failed” config?

joyqat on Mar 1, 2019

I’m also having this issue and I’ve tried multiple ways of trying to fix it…

Gitea in a docker container at home
Gitea running normally at home
Gitea on Docker on a VPS
Gitea running normally on a VPS
Both a clean database and imported database

I get the same i/o timeout issue no matter what I do. Of course webhooks are quite important especially with CI/CD. I was hoping to move to Gitea this weekend but after spending hours trying to get this work, I’m not sure now lol

Edit: using the latest stable version of Gitea btw which is 1.9.4

Another edit an hour later: I’ve sorted it. Turns out Drone checks if there’s a .drone.yml file and if there isn’t one, it won’t allow a webhook? Very odd but glad I’ve sorted it!

deanpcmad on Oct 27, 2019

same problem…

I have set DELIVER_TIMEOUT=30 and still timeout.
I execute curl <webhook URL> command inside my gitea container, it’s very fast and no problem.
I test webhook within another repository, it’s worked. This repository’s webhook URL is same with previous repository’s webhook URL, all they are my Drone CI adress. How can it time out?

Veitor on Mar 3, 2020

I’m also having this issue and I’ve tried multiple ways of trying to fix it…

Gitea in a docker container at home

Gitea running normally at home

Gitea on Docker on a VPS

Gitea running normally on a VPS

Both a clean database and imported database

I get the same i/o timeout issue no matter what I do. Of course webhooks are quite important especially with CI/CD. I was hoping to move to Gitea this weekend but after spending hours trying to get this work, I’m not sure now lol

Edit: using the latest stable version of Gitea btw which is 1.9.4

Another edit an hour later: I’ve sorted it. Turns out Drone checks if there’s a .drone.yml file and if there isn’t one, it won’t allow a webhook? Very odd but glad I’ve sorted it!

Did you tried playing with the DELIVER_TIMEOUT parameter in the config? It helped me to fix this issue.

c0h1b4 on Oct 28, 2019

I think the timeout is already configurable, @barryp. It’s this in custom/app.ini:

[webhook]
DELIVER_TIMEOUT = 60

…60 is the value I have set. If I try the “test” delivery, it does work fine some of the time. If I try it manually with curl from the same host, it always works. I think it’s definitely a bug in Gitea.

sbrl on Jan 23, 2019