gitea: Coredump after pushing new branch with an invalid webhook (RasbperryPi 3)

  • Gitea version (or commit ref): 1.11.4
  • Git version: 2.26.2
  • Operating system: archlinuxARM ($uname -srmo : Linux 5.6.7-1-ARCH aarch64 GNU/Linux)
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • MSSQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist: https://pastebin.com/0PGUayjD

Description

I run a private gitea instance with the systemd service of the archlinux package. I am the only user on this instance. I have I think 6 quite small repos. I locally created a new branch in a repo. I pushed it by executing : git push origin {BranchName} Gitea has been restarting every 20 seconds since, and generating tons of coredumps (following longer intervals are due to reboots and config tweaks that did not help) :

$ sudo coredumpctl
TIME                            PID   UID   GID SIG COREFILE  EXE
Mon 2020-04-27 23:59:57 CEST    856   973   973  31 missing   /usr/bin/gitea
Tue 2020-04-28 00:01:51 CEST    958   973   973  31 missing   /usr/bin/gitea
Tue 2020-04-28 00:09:17 CEST   1139   973   973  31 missing   /usr/bin/gitea
Tue 2020-04-28 00:25:34 CEST   1344   973   973  31 missing   /usr/bin/gitea
Tue 2020-04-28 00:25:55 CEST   1375   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 00:58:50 CEST   1046   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 00:58:59 CEST   1074   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:07:54 CEST   1174   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:08:47 CEST   1212   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:10:03 CEST   1257   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:13:44 CEST   1288   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:13:54 CEST   1322   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:20:49 CEST   1384   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:21:38 CEST   1419   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:56:11 CEST   2937   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 01:59:21 CEST   2964   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 02:01:40 CEST   2994   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 02:02:17 CEST   3017   973   973  31 present   /usr/bin/gitea
Tue 2020-04-28 02:02:29 CEST   3041   973   973  31 present   /usr/bin/gitea

I tried to gdb the coredumps but I think its output is not very useful : (sorry, this is the first time I ever launch gdb, I may not be very aware of what’s useful or not. Saw it in another issue …)

$ sudo coredumpctl gdb 3017
           PID: 3017 (gitea)
           UID: 973 (gitea)
           GID: 973 (gitea)
        Signal: 31 (SYS)
     Timestamp: Tue 2020-04-28 01:58:01 CEST (29min ago)
  Command Line: /usr/bin/gitea web -c /etc/gitea/app.ini
    Executable: /usr/bin/gitea
 Control Group: /system.slice/gitea.service
          Unit: gitea.service
         Slice: system.slice
       Boot ID: b2c2901b20dd4532adcc64fe2f8f5ef3
    Machine ID: e9924b3339eb455fbe8473f0fb3250c3
      Hostname: rpi3
       Storage: /var/lib/systemd/coredump/core.gitea.973.b2c2901b20dd4532adcc64fe2f8f5ef3.3017.1588031881000000000000.lz4
       Message: Process 3017 (gitea) of user 973 dumped core.
                
                Stack trace of thread 3033:
                #0  0x0000aaaadb4ffcd0 n/a (gitea + 0x9e5cd0)
                #1  0x0000aaaadb4fe224 n/a (gitea + 0x9e4224)
                #2  0x0000aaaadb595fc4 n/a (gitea + 0xa7bfc4)
                #3  0x0000aaaadb5a8d40 n/a (gitea + 0xa8ed40)
                #4  0x0000aaaadb4cc1c4 n/a (gitea + 0x9b21c4)
                #5  0x0000aaaadb597b70 n/a (gitea + 0xa7db70)
                #6  0x0000aaaadb5a6054 n/a (gitea + 0xa8c054)
                #7  0x0000aaaadb5a6710 n/a (gitea + 0xa8c710)
                #8  0x0000aaaadb4b40a4 n/a (gitea + 0x99a0a4)
GNU gdb (GDB) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/gitea...
(No debugging symbols found in /usr/bin/gitea)
[New LWP 3033]
[New LWP 3019]
[New LWP 3020]
[New LWP 3023]
[New LWP 3022]
[New LWP 3021]
[New LWP 3024]
[New LWP 3017]
[New LWP 3025]
[New LWP 3026]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/bin/gitea web -c /etc/gitea/app.ini'.
Program terminated with signal SIGSYS, Bad system call.
#0  0x0000aaaadb4ffcd0 in ?? ()
[Current thread is 1 (Thread 0xffff6ffff1a0 (LWP 3033))]
(gdb) bt
#0  0x0000aaaadb4ffcd0 in ?? ()
#1  0x0000000000000007 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

I thinks something broke during the push command, and now gitea is trying to redo this at boot and constantly failing ? If I can give you some more info, just ask !

Screenshots

Not relevant

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 19 (6 by maintainers)

Most upvoted comments

Please see if https://github.com/go-gitea/gitea/issues/9792#issuecomment-691291999 solves the issue for you as well.

+1
ltguillaume on Sep 11, 2020

Gitea was receiving a 404 response from my non existing drone instance adress.

+1
Ouack23 on May 4, 2020

I finally found it !!

Additional info :

  • I use the ssh server from the OpenSSH system package
  • my ssh server is listening to port 2223 instead of the default 22.
  • I use gitea with socket in /run/gitea/gitea.socket

First, after a coredump, running sudo -u gitea gdb --args gitea web -c /etc/gitea/app.ini, running the debugger, stopping gdb and restarting the service works : the gitea instance starts successfully and the pushed commit is found in the gitea repo. However, it is not listed in the activity dashboard.

I created a test repository and :

  • committed a file in master -> OK
  • committed a file in a new branch -> OK

So then I asked myself what the hell is the difference between my “coredump” repo and this one ??

The reason why this repo coredumped after commits is an invalid webhook : I hosted a while ago a drone instance on another raspberry and linked it to my gitea instance, but it took too much resources, so I temporarily disabled it. A invalid webhook was still enabled on this repository (with a red cross next to it in the {repo}/settings/hooks page). After deleting it, no more coredumps happen. Why was it working in the master branch ? I don’t know.

As the manual gitea command seems to work, I think systemd hardening is wrong somewhere, regarding webhooks. I join my gitea systemd setup :

/etc/systemd/system/multi-user.target.wants/gitea.service
===========================================
[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target
After=mysqld.service
After=postgresql.service
After=memcached.service
After=redis.service

[Service]
User=gitea
Group=gitea
Type=simple
WorkingDirectory=~
RuntimeDirectory=gitea
LogsDirectory=gitea
StateDirectory=gitea
Environment=USER=gitea HOME=/var/lib/gitea GITEA_WORK_DIR=/var/lib/gitea
ExecStart=/usr/bin/gitea web -c /etc/gitea/app.ini
Restart=always
RestartSec=2s
CapabilityBoundingSet=
NoNewPrivileges=True
PrivateUsers=true
PrivateDevices=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ProtectControlGroups=yes
ProtectKernelTunables=true
ProtectKernelModules=yes
ReadWritePaths=/etc/gitea/app.ini
LockPersonality=true
MemoryDenyWriteExecute=true
RestrictRealtime=true
SystemCallArchitectures=native
SystemCallFilter=@system-service

[Install]
WantedBy=multi-user.target

/etc/systemd/system/gitea.service.d/override.conf
=====================================
[Service]
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
PrivateUsers=false
Restart=no
ReadWritePaths=/etc/gitea/app.ini
ReadWritePaths=/var/lib/gitea

Do you have any idea on your end about why would this happen ?

+1
Ouack23 on May 2, 2020
Read more comments on GitHub
← gitea: Conflicting files shown on pull request but merging without conflict
gitea: Could not read your security key →