containerd: 1.5.10 causes memory leak in mysql container; regression from 1.4.13

So to summarize, this seemingly helped me on Fedora 37

sudo nano /lib/systemd/system/containerd.service
LimitNOFILE=INFINITY --> LimitNOFILE=1048576 
docker stop <db>
docker rm <db>
sudo systemctl daemon-reload
sudo systemctl restart containerd
docker-compose up -d

I installed 1.5 and confirmed problem still persisted. I then modified /lib/systemd/system/containerd.service to read LimitNOFILE=1048576. I then restarted the service and the daemon: sudo systemctl restart service containerd sudo systemctl daemon-reload. Problem still persisted. I then restarted the service once more, because I thought the daemon might have cached the service (so in total I restarted service, daemon, service) and the problem was gone. Docker stable repos don’t have 1.6 available so I didn’t test that.

Just to give this topic a little push here. I had the same problem. In our docker-compose file we need 3 different mysql servers and as they had no memory limits set they killed my whole system (fedora 37). A simple docker-compose up led to a system freeze and only a hard reset helped.

The change in /lib/systemd/system/containerd.service by setting the LimitNOFILE=1048576 helped!

What worked for me on Fedora 36: sudo vim /lib/systemd/system/containerd.service ->

#LimitNOFILE=infinity
LimitNOFILE=1048576

Then:

systemctl daemon-reload
sudo systemctl restart containerd
docker ps -a # (find id and then docker rm id)

run again: docker run --name=mysql1 -d mysql/mysql-server:5.7

After that it worked for me, where with the value set to unlimited, the container crashed in seconds. This change does not seem to be needed on Ubuntu and derivates, is there anything else messing this up?

Seems there is a bug in Systemd, if you need more than 65535 files, need to change the LimitNOFILE manually to a bigger number, such as 1048576

https://github.com/systemd/systemd/issues/6559

Maybe we need to revert this PR 4475

If this is an issue of resource allocation, I suppose that would resolve it.

I haven’t tried to reproduce the mysql one, but it’s been a problem that affected many software where outside of containers the expected limits are considerably lower.

Default is 1024 soft limit. docker.service and containerd.service have overrided that to infinity which can be as high as over 1 billion. Usually that just stalls the software for a much longer duration (sometimes hours), but some software affected by this may also do some allocation, which as you can guess 1e3 vs 1e9 is quite the delta, a million times more. So if your software would normally allocate 1MB in this scenario and that allocation code was affected by this much large limit increase, it’d now use 1TB.

Normally it’s just CPU from tasks like iterating through the limit range and closing potentially (highly unlikely) open file descriptors.

If this is actually a memory leak or some bit of bad code, it might be worthwhile to dive into the two versions noted here and see if the code causing the issue can be identified.

True. In some projects affected that I used, I helped track down the problem area and implement a workaround. However outside of container environments where the limits are sane and typically only raised when necessary, you’re not likely to encounter this sort of problem. The bug is more to do with misconfiguration with LimitNOFILE=infinity being applied to the containers (and each process in those containers being run with a soft limit well above the expected 1024).

In the meantime, you can start the container with docker run --ulimit "nofile=1024:524288" extra option, or with systemd services set a drop-in override with systemctl edit containerd.service with the following:

[Service]
LimitNOFILE=1024:524288

• You may need to do the same for docker.service until a release is out with that updated too.
• After applying the edit for these files, restart Docker systemctl restart docker.
• You can verify these are the new limits applied with docker run --rm -it alpine ash -c 'ulimit -Sn && ulimit -Hn', you should get 1024 and 524288.

Just a heads-up that the LimitNOFILE=infinity setting in both docker.service and containerd.service files has finally been removed from their respective projects.

Possibly those changes will be part of new releases before the end of the year:

• v25 for Docker (with backports planned to 20.10)
• v1.7.8 v2.0 for containerd?

After that happens, this issue can be considered resolved?

I’m having the same issue on Manjaro (Kernel 6.1.12-1), and changing LimitNOFILE=1048576 change made it work again.

Launching mysql:8.* works great, but mysql:5.7.* causes immediate 100% memory consumption (htop), and results in the following in /var/log/messages:

MySQL is having https://bugs.mysql.com/bug.php?id=96525.

I wanted to add my own experience with this issue: apparently I can only reproduce it if swap is enabled. Starting the container with no swap (a.k.a running swapoff <device>) makes the container work just fine, and I can then reenable swap later

@ThatCoffeeGuy thank you! This solved the same problem in my distro ( Manjaro linux )!

FYI this also breaks cups.

Hello. I’m experiencing the same issue.

Launching mysql:8.* works great, but mysql:5.7.* causes immediate 100% memory consumption (htop), and results in the following in /var/log/messages:

2022-06-06T17:23:24.094275-04:00 laptop kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-xxx.scope,task=mysqld,pid=38421,uid=0
2022-06-06T17:23:24.094288-04:00 laptop kernel: Out of memory: Killed process 38421 (mysqld) total-vm:16829404kB, anon-rss:12304300kB, file-rss:108kB, shmem-rss:0kB, UID:0 pgtables:28428kB oom_score_adj:0
2022-06-06T17:23:24.094313-04:00 laptop systemd[1]: docker-xxx.scope: A process of this unit has been killed by the OOM killer.
2022-06-06T17:23:24.856029-04:00 laptop systemd[1]: docker-xxx.scope: Deactivated successfully.

Versions:

Docker version 20.10.16, build aa7e414
containerd containerd.io 1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
runc version 1.1.1
commit: v1.1.1-0-g52de29d
spec: 1.0.2-dev
go: go1.17.9
libseccomp: 2.5.3
Linux laptop1138 5.17.12-200.fc35.x86_64 #1 SMP PREEMPT Mon May 30 16:58:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Fedora release 35 (Thirty Five)

Limits:

cat /proc/$(pidof dockerd)/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            1073741816           1073741816           files     
Max locked memory         8388608              8388608              bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       62780                62780                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 62780
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 64000
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 62780
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

systemctl show containerd | grep LimitNOFILE
LimitNOFILE=infinity
LimitNOFILESoft=infinity

Creating the following custom limit for containerd does not resolve the issue. /etc/systemd/system/containerd.service.d/custom.conf

[Service]
LimitNOFILE=1048576

Note: Prior systemctl show containerd | grep LimitNOFILE would report ‘unlimited’.

systemctl show containerd | grep LimitNOFILE
LimitNOFILE=1048576
LimitNOFILESoft=1048576

This change did not resolve the issue as suggested here: https://github.com/containerd/containerd/issues/3201

I think you should do daemon-reload first and then restart containerd. And it only makes sure that it works for new containers, not for existing ones.

I have the same issue

runc version 1.0.3
commit: v1.0.3-0-gf46b6ba
spec: 1.0.2-dev
go: go1.16.15
libseccomp: 2.5.3

I implemented a temp solution using ulimit like this comment. It’s worked for me. docker-library/mysql#579 (comment)

Yes, can confirm that this is a valid workaround. It seems very likely that the increase to infinity nofile size is the problem on some systems.

I have the same issue

runc version 1.0.3
commit: v1.0.3-0-gf46b6ba
spec: 1.0.2-dev
go: go1.16.15
libseccomp: 2.5.3

I implemented a temp solution using ulimit like this comment. It’s worked for me. https://github.com/docker-library/mysql/issues/579#issuecomment-1074069882