memcached: Memory leak?

Hi,

Experiencing an issue with the memory usage. The memcached process resident size is significantly larger than the memory allocated to the slabs. As it only grows, it apparently gets killed by OOM eventually (usually couple months in current setup) STAT limit_maxbytes 23068672000 doesn’t actually help, as the RSS increases independently.

Memcached is exclusively running on a number of aws r4.xlarge instances with ~1.5k get/s over ~80k persistent tcp connections. Non-default parameters: -m 22000 -t 4

3 days uptime process (~7G RSS vs 3.5 of alloced):

$ echo stats | nc localhost 11211|egrep ' bytes '; echo 'stats slabs' | nc localhost 11211|egrep total_malloced; egrep -i rss /proc/`pidof memcached`/status ; echo -n Anon: ;sudo pmap -xx `pidof memcached`|egrep anon | awk 'BEGIN{s=0}{s+=$3} END{print s}'
STAT bytes 2452758519
STAT total_malloced 3687841792
VmRSS:   7003412 kB
Anon:7003180

soon-to-be-killed (process running >2mo):

$ echo stats | nc localhost 11211|egrep ' bytes '; echo 'stats slabs' | nc localhost 11211|egrep total_malloced; egrep -i rss /proc/`pidof memcached`/status ; echo -n Anon: ;sudo pmap -xx `pidof memcached`|egrep anon | awk 'BEGIN{s=0}{s+=$3} END{print s}'
STAT bytes 7909043757
STAT total_malloced 23068630896
VmRSS:  30135748 kB
Anon:30135880

It’s reproducing on both legacy 1.4.14 ubuntu trusty version and latest compiled 1.5.7.

Any ideas?

Thanks.

memcached.stats.txt