mopidy: Apparent memory leak during normal MP3 playback

I have what appears to be a memory leak on a Mopidy installation on a RasPi, booting and running over NFS. Initially I thought the problem was that I had no swap set up, so I added some, but it continues to happen.

The extremely odd thing about this is that the NFS root image is identical to three other RasPi servers handling sound systems in other rooms of the house, but those other three systems have run for weeks without exhibiting any issues. The failing unit freezes up nearly once a day. The only difference among all four servers is the exact content of the music library, but all libraries are pretty small…less than 2k songs, far less in some cases. All songs are in MP3 format, in case it makes a difference.

Here’s a screenshot of top that I had left running to see the system state when it freezes. Note the insane load and 0k free swap, and the fact that kswapd is pegging the CPU, but also that mopidy is the one using all the memory. Normally the mopidy process is down around 5-6% mem usage.

I’m already using a local SQLite db. I can run a local scan from the command line with no issues.

After the above screenshot, I power-cycled the RasPi and got back into a top session so I could pay more attention to it. The %MEM value reported has slowly crept upward from about 5% to 42%, where it is now. I expect it will continue to rise if I do nothing.

In contrast, one of the stable RasPi audio servers has been running for over two weeks solid and playing audio for at least 72 hours without stopping, and the mopidy process is at 6.7% usage. It even has a slightly larger library.

…and after a while longer, the memory use did indeed continue to creep upwards (at 50% when I acted), and I ran sudo service mopidy force-reload as a test to see if that would help. It did–mopidy memory usage dropped down near zero for a moment, and then back to the expected ~5%. No full service restart needed, though I assume that also would have done the same.

The different RasPis involved all report Raspberry Pi 3 Model B Rev 1.2 from the output of cat /sys/firmware/devicetree/base/model. They are running over NFS mounted root folders that are exact copies of each other, except for the hostname. The contents of /etc/mopidy/mopidy.conf are identical. The memory usage takes many hours to grow significantly, but it will eventually eat up all available RAM and swap space.

I have confirmed as well that the memory use stops increasing when audio is stopped (not sure about paused though). The following set of graphs shows RAM and SWAP usage on the problem system over a 24-hour period, including two full PLAY-STOP cycles (hours long) with one final PLAY event at the end to confirm. I did another force-reload immediately after this to avoid a complete system freeze and drop RAM/swap usage back down to normal levels.

You can see how the RAM is eaten up first (from 50% to 80%) during the first cycle, then shortly after the second cycle starts, it eats the last bit of safe RAM (from 80% to 85%) and then switches to consuming swap (from 20% to 80%) until stopped. Finally, it ticks up again at a consistent rate about 15 minutes before the end of the graph, which is where I started playback again.

Note, as far as I can tell, pausing playback in the Mopidy/moped web interface has the same effect as stopping it in this regard, i.e. it stops eating memory.

Host platform is Raspbian on a RasPi 3B v1.2:

Linux lucas-audio 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux

It is possible that this issue is related to either #1750 (most likely) or #1648 (less likely).