containerpilot: memory leak

I was using ContainerPilot v3.3.0 and found the ContainerPilot process memory leak.

Linux 192-168-161-100 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux ubuntu 16.04 docker image: phusion/baseimage:0.9.18

ContainerPilot configuration:

{
  consul: "127.0.0.1:8500",
  jobs: [
    {
      name: "p-job",
      port: 80,
      exec: "p-job start",
      restarts: "unlimited",
      when: {
        source: "generate-file",
        once: "exitSuccess",
      },
      health: {
        exec: "curl 127.0.0.1:80",
        interval: 2,
        ttl: 10,
        timeout: "10s"
      },
    },
    {
      name: "consul-client",
      exec: [
        "consul",
        "agent",
        "-config-dir=/etc/consulClient.conf",
        "-data-dir=/var/lib/consul",
        "-retry-join=consul-server",
        "-disable-host-node-id"
      ],
      health: {
        exec: "consul operator raft -list-peers | grep leader",
        interval: 2,
        ttl: 10,
        timeout: "10s"
      },
      restarts: "unlimited"
    },
    {
      name: "generate-file",
      exec: [
        "consul-template",
        "-once",
        "-consul-addr",
        "127.0.0.1:8500",
        "-template",
        "/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json"
      ],
      when: {
        source: "consul-client",
        once: "healthy"
      }
    },
    {
      name: "consul-leave",
      exec: "consul leave",
      when: {
        source: "consul-client",
        once: "stopping"
      }
    },
    {
      name: "regenerate-file-a",
      exec: [
        "consul-template",
        "-once",
        "-consul-addr",
        "127.0.0.1:8500",
        "-template",
        "/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json"
      ],
      when: {
        source: "watch.a",
        each: "changed"
      }
    },
    {
      name: "regenerate-file-b",
      exec: [
        "consul-template",
        "-once",
        "-consul-addr",
        "127.0.0.1:8500",
        "-template",
        "/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json"
      ],
      when: {
        source: "watch.b",
        each: "changed"
      }
    },
    {
      name: "regenerate-file-c",
      exec: [
        "consul-template",
        "-once",
        "-consul-addr",
        "127.0.0.1:8500",
        "-template",
        "/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json"
      ],
      when: {
        source: "watch.c",
        each: "changed"
      }
    },
    {
      name: "send-sighup-a",
      exec: "kill -HUP ${p-job PID}",
      when: {
        source: "regenerate-file-a",
        each: "exitSuccess"
      }
    },
    {
      name: "send-sighup-b",
      exec: "kill -HUP ${p-job PID}",
      when: {
        source: "regenerate-file-b",
        each: "exitSuccess"
      }
    },
    {
      name: "send-sighup-c",
      exec: "kill -HUP ${p-job PID}",
      when: {
        source: "regenerate-file-c",
        each: "exitSuccess"
      }
    },
  ],
  watches: [
    {
      name: "a",
      interval: 1
    },
    {
      name: "b",
      interval: 1
    },
    {
      name: "c",
      interval: 1
    },
    
  ],
  control: {
    socket: "/var/run/containerpilot.socket"
  }
}

pmap -d ${containerpilot PID}

Address           Kbytes Mode  Offset           Device    Mapping
0000000000400000    3788 r-x-- 0000000000000000 000:000bc containerpilot
00000000007b3000    2868 r---- 00000000003b3000 000:000bc containerpilot
0000000000a80000     216 rw--- 0000000000680000 000:000bc containerpilot
0000000000ab6000     140 rw--- 0000000000000000 000:00000   [ anon ]
000000c000000000      56 rw--- 0000000000000000 000:00000   [ anon ]
000000c41fe58000    1696 rw--- 0000000000000000 000:00000   [ anon ]
000000c420000000    7168 rw--- 0000000000000000 000:00000   [ anon ]
000000c420700000    3072 rw--- 0000000000000000 000:00000   [ anon ]
000000c420a00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c420b00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c420c00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c420d00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c420e00000    2048 rw--- 0000000000000000 000:00000   [ anon ]
000000c421000000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421100000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421200000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421300000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421400000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421500000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421600000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421700000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421800000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421900000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421a00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421b00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421c00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421d00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421e00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c421f00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422000000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422100000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422200000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422300000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422400000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422500000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422600000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422700000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422800000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422900000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422a00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422b00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422c00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422d00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422e00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c422f00000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c423000000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c423100000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c423200000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c423300000    1024 rw--- 0000000000000000 000:00000   [ anon ]
000000c423400000    1024 rw--- 0000000000000000 000:00000   [ anon ]
00007fa8b9817000    3584 rw--- 0000000000000000 000:00000   [ anon ]
00007ffee4d07000     136 rw--- 0000000000000000 000:00000   [ stack ]
00007ffee4da6000       8 r---- 0000000000000000 000:00000   [ anon ]
00007ffee4da8000       8 r-x-- 0000000000000000 000:00000   [ anon ]
ffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]
mapped: 66776K    writeable/private: 60100K    shared: 0K

dump and view ContainerPilot memory, the memory stores a large number of duplicate the following:

  • container environment variable
  • a, b, c services registered to consul information
  • Send-sighup event

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 37 (32 by maintainers)

Most upvoted comments

We can confirm this bug has been fixed, no obvious leak after 16 hours running, thanks!

Sorry for the delay, @JeromeGe is taking a vacation right now, so I’m arranging anther colleague to provide a reproduce demo, will submit today, thanks for your help 😃