portainer: Memory leak since upgrade to 1.19.1

BUG REPORT INFORMATION

Description

upgrading to 1.19.1 (from 1.18.1) caused portainer to start leaking memory

the upgrade moment can be seen here (monitored via influxdb/telegraf):

portainer_leak

redeployed again to make sure. After 1 day:

portainer_leak2

Steps to reproduce the issue:

deploy

docker run --restart=always -d --name portainer -p 9000:9000 \
                  -v /portainer/data:/data \
                  -v /var/run/docker.sock:/var/run/docker.sock \
                  portainer/portainer

monitor & wait for sufficient amount of time to observe the leak

Could this be a goroutine leak by any chance?

Technical details:

Portainer version: 1.19.1
Target Docker version (the host/cluster you manage):

 Engine:
  Version:      18.03.0
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Fri May  4 12:40:05 2018
  OS/Arch:      linux/amd64
  Experimental: false

Platform (windows/linux): Photon OS
Command used to start Portainer: see above

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 29 (10 by maintainers)

Most upvoted comments

Looks good here also…

grafana - docker 2018-09-04 17-39-48

Codelica on Sep 4, 2018

Would also like to chime in and say thanks, I pulled this image about 12 hours ago and haven’t seen any leaking since. Anxiously waiting for this in the latest image 🙂

SapphicCode on Sep 4, 2018

Awesome @d-led, I’ll merge the fix.

Thanks a lot for the feedback !

deviantony on Sep 4, 2018

@deviantony yep

6f6d2d412365 portainer/portainer:fix1991-open-files  "/portainer"  4 hours ago

the red line is the moment where I deployed. Stil looks ok

d-led on Sep 3, 2018

@deviantony yes, no leaks every 5 mins. Strange, that contrary to @neothematrix, I could pass the flag (as seen in the UI) to stop snapshots, but the flag did not have an effect

file handles after setting the interval to 9999h:

d-led on Aug 24, 2018

in my case, turning off snapshots (I managed to pass the --no-snapshot flag) seems to have completely solved the filedescriptor leak, latest deploy of the portainer container was at 18:00 in the graph:

however I never noticed the memory leak issue:

however I did change another parameter, I’m now bind mounting /var/run/docker.sock instead of the whole /var/run folder as I was doing before (I had no choice with Container Station since it allowed only directory bind mounts, not files), I’ll try to perform a reverse-test by removing the --no-snapshot flag and see if the fd leak starts again.

neothematrix on Aug 24, 2018

@deviantony thanks too! Portainer rocks 😄

d-led on Aug 23, 2018