ModSecurity: memory leak on nginx reload

RAM usage constantly grows on nginx -s reload

Having modsecurity rules loaded (even with modsecurity off) causes RAM usage to grow with each nginx -s reload and ultimately leads nginx to stuck with messages like:

Logs and dumps (/var/log/nginx/error.log)

Output of: 2020/08/06 20:00:20 [alert] 1962#1962: fork() failed while spawning “worker process” (12: Cannot allocate memory) 2020/08/06 20:00:20 [alert] 1962#1962: sendmsg() failed (9: Bad file descriptor) 2020/08/06 20:00:20 [alert] 1962#1962: fork() failed while spawning “worker process” (12: Cannot allocate memory) 2020/08/06 20:00:20 [alert] 1962#1962: sendmsg() failed (9: Bad file descriptor) 2020/08/06 20:00:20 [alert] 1962#1962: fork() failed while spawning “cache manager process” (12: Cannot allocate memory) 2020/08/06 20:00:20 [alert] 1962#1962: sendmsg() failed (9: Bad file descriptor)

To Reproduce

1. Configure nginx to load rules: /etc/nginx/nginx.conf

http {
...
   modsecurity off;
   modsecurity_rules_file /etc/nginx/modsec/main.conf;
..
}

1. Restart Nginx and check rules were loaded (/var/log/nginx/error.log):

2020/08/06 08:57:13 [notice] 13627#13627: ModSecurity-nginx v1.0.1 (rules loaded inline/local/remote: 0/911/0)

1. Generate load:

./nikto.pl -h https://your-site-name

1. Run several ‘nginx -s reload’ (with 3-4 minutes interval) and check RAM consumption with free -m command before and after nginx reload:

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         433        2122          30        1395        3136
Swap:          2043          49        1994

# nginx -s reload

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         451        2103          30        1395        3117
Swap:          2043          49        1994

# nginx -s reload

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         464        2083          30        1404        3104
Swap:          2043          49        1994

# nginx -s reload

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         481        2051          30        1417        3086
Swap:          2043          49        1994

.....

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3951         901        1534          30        1515        2666
Swap:  

Expected behavior

‘RAM used’ should not steadily grow and should stay around the same level as it does for example without modsecurity rules loaded (in which case ‘ram used’ stays about 300 MB)

Server

• ModSecurity v3 master - https://github.com/SpiderLabs/ModSecurity/commit/51d06d7a8edc7a400782c06e24ebe71834736f77 with nginx-connector v1.0.1
• WebServer: nginx-1.18.0
• OS : Ubuntu 16.04

Rule Set:

• https://github.com/coreruleset/coreruleset/archive/v3.3.0.tar.gz

Additional context

The same happens with modsecurity on in server’s context. Using SecResponseBodyAccess Off in modsecurity.conf