gitea: Gitea keeping taking 100% CPU usage, and runs slowly in git pushing

  • Gitea version (or commit ref): 1.12.0+dev-440-gf5a20250a

  • Git version: git version 2.20.1

  • Operating system: Raspbian GNU/Linux 10 (buster) on Raspberry Pi zero w

  • Database (use [x]):

    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:

    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist:

  • Architecture: linux-arm-6

Description

After upgrading the gitea to this version, I found that gitea runs very slow. By checking the CPU usage, the CPU usage keeps 100%. And I checked the ps, found that seems the gitea hook processes (2395, 964) are taking up lots resource, I would like to know what are they and how can I track them, and how to stop them. As far as I know, there is no git hook setting in my repo that runs on this server. So I think this could be the gitea’s system hook. And I see PID 2366 saying duchenpaul/homebridge-docker.git is executing hook, but there is no hook script set in that repo As the PID 961, I dont know what is going on there.

I checked the log, only the sql that queried in the log, no errors

  928 ?        Ss     0:00  \_ sshd: git [priv]
  937 ?        S      0:00  |   \_ sshd: git@notty
  939 ?        Rsl   14:49  |       \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
  987 ?        Ss     0:00  \_ sshd: git [priv]
  996 ?        S      0:00  |   \_ sshd: git@notty
  998 ?        Rsl   12:43  |       \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
 1310 ?        Ss     0:00  \_ sshd: git [priv]
 1319 ?        S      0:00  |   \_ sshd: git@notty
 1321 ?        Rsl   10:04  |       \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
 2062 ?        Ss     0:00  \_ sshd: pi [priv]
 2071 ?        S      0:02  |   \_ sshd: pi@pts/1
 2074 pts/1    Ss     0:00  |       \_ -bash
 2482 pts/1    S+     0:00  |           \_ vi gitea.log
 2349 ?        Ss     0:00  \_ sshd: git [priv]
 2358 ?        S      0:00      \_ sshd: git@notty
 2360 ?        Ssl    0:03          \_ /home/git/gitea/gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini
 2366 ?        Sl     0:00              \_ git-receive-pack duchenpaul/homebridge-docker.git
 2390 ?        S      0:00                  \_ bash hooks/post-receive
 2394 ?        S      0:00                      \_ bash ./hooks/post-receive.d/gitea
 2395 ?        Rl     4:55                          \_ /home/git/gitea/gitea hook --config=/home/git/gitea/custom/conf/app.ini post-receive
  454 tty1     Ss+    0:00 /sbin/agetty -o -p -- \u --noclear tty1 linux
  525 ?        Ss     0:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
  619 ?        S      0:00  \_ php-fpm: pool www
  620 ?        S      0:00  \_ php-fpm: pool www
  586 ?        Ss     0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
  590 ?        S      0:00  \_ nginx: worker process
  654 ?        Ss     0:00 /lib/systemd/systemd --user
  680 ?        S      0:00  \_ (sd-pam)
  700 ?        Ss     0:00 /lib/systemd/systemd --user
  723 ?        S      0:00  \_ (sd-pam)
  753 ?        Ss     0:00 /usr/sbin/exim4 -bd -q30m
  961 ?        S      0:00 bash hooks/update refs/heads/master 2b1fb9ae3617e7fc0b9b496897b542e1e3e24375 9a610f1bb0fdd659a4fa5ef7576ff3dc27690f22
  963 ?        S      0:00  \_ bash ./hooks/update.d/gitea refs/heads/master 2b1fb9ae3617e7fc0b9b496897b542e1e3e24375 9a610f1bb0fdd659a4fa5ef7576ff3dc27690f22
  964 ?        Rl    14:45      \_ /home/git/gitea/gitea hook --config=/home/git/gitea/custom/conf/app.ini update refs/heads/master 2b1fb9ae3617e7fc0b9b496897b542e1e3e24375 9a610f1bb0fdd659a4fa5ef7576ff3dc27690f22
 2175 ?        Ssl    1:49 /home/git/gitea/gitea web

[Update] Summarize

  • The hanging of gitea serv key-3 --config=/home/git/gitea/custom/conf/app.ini is introduced from git client’s requests, for some reason, some git request did not finishes properly, instead, taking up all the CPU for hours and slow down the system.
  • And these process does not related to the gitea main service, even i stops the gitea service, these process is still hanging there.
  • It looks the bug started from v1.11.2 (0768823) in the release, I am using v1.11.1 (ff24f81), it is ok, but as long as I switch to v1.11.2, above thing start to happen in a short time.

Screenshots

image

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 53 (21 by maintainers)

Commits related to this issue

Most upvoted comments

#10684 may fix this, it will be released on v1.11.3

I compiled 1.11.2 with go 1.13.8. It seems OK now.

Yes, I noticed there is issue with version number. But I am sure there is issue in new version. I download the gitea from the master folder here: https://dl.gitea.io/gitea/master/gitea-master-linux-arm-6, this is the newest version and where this thing occurs.

Thank you for your replying, I did a reboot again, looks ok for now. I didn’t cancel any transactions in-fight, from what I see for client side, all transactions are finished normally. I believe PID 964 I referring to is blocked due to the high CPU usage of ssh transactions

My questions are:

  • Is there are timeout settings that cancel the in-fight ssh transactions from server side which taking too much time?
  • And is there a way to track back what is going on with those ssh transactions.

anyway, I will keep monitoring for 3 days, if it is ok, I will close this issue.

This gitea server is only serving myself, I think it still is able to handle the job well. 😄 image

output for vmstat 3 999

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0   4036  68112  28512 141908    0    5   523    45  576 1210 17  9 72  2  0
 0  0   4036  68112  28512 141908    0    0     0     0  434  877  4  3 93  0  0
 0  0   4036  68112  28512 141908    0    0     0     0  405  856  6  1 93  0  0
 1  0   4036  68112  28512 141908    0    0     0     0  454  917  4  2 94  0  0
 3  0   4036  68112  28512 141908    0    0     0     0  532 1086  7  2 91  0  0