flexx: Memory leak in PyWidget

Looks like the PyWidget never gets garbage-collected. Causing all its attributes to stick around too. When these contain e.g. large datasets, this can cause severe memory leaks.

This needs some research to see what is holding a ref, and if we can solve this using e.g. a weakref. See a self-contained test-example further down.

original post

Hi @almarklein,

I have a flx.PyWidget that has flx.Widget that wraps a javascript widget as a member:

class MyWrappedWidget(flx.Widget):
  ...
  def dispose(self):
    # clean up code
    print("clean-up-completed")

  @flx.action
  def manual_dispose(self):
    # clean up code
    print("manual-clean-up-completed")

class MyPyWidget(flx.PyWidget):
  def init(self):
    with flx.VFix(flex=1) as self.frame:
      self.mywidget = MyWrappedWidget() # the wrapped widget defined above

  def dispose(self):
    # self.mywidget.dispose()
    self.mywidget.manual_dispose()
    print("MyPyWidget-clean-up")

Now when the flx.PyWidget is disposed, the manual_dispose call seems to be made, but doesn’t execute, the print is never made to the command line. I also tried calling the flx.Widget’s dispose method directly, but it’s also doesn’t execute. The wrapped flx.Widget (MyWrappedWidget) can be quite large (100’s of MBs), so when old ones should be deleted and new ones created, the memory just stacks up, it doesn’t get released. That’s how it seems to me, don’t hold me to this, I may be wrong.

Is there a way to force the child flx.Widget to get disposed?

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 26 (26 by maintainers)

Most upvoted comments

Also made a new release. Thanks @matkuki for your help in fixing this!

almarklein on Apr 12, 2022

It’s funny, because that change alone, does not fix things for me. So let’s do both. #725

almarklein on Apr 12, 2022

mmm, interesting that it behaves differently from what I see.

Could you try changing that “suspicious code” that I mentioned above to:

    def __del__(self): 
        if not self._disposed: 
           self._dispose()

and see if that helps? It’s a bit of a long shot, but I don’t know what else to try right now 😕

almarklein on Apr 12, 2022

Thanks! I’ll try to have a look soon. I also updated the title and the top post to describe the issue as we now understand it.

almarklein on Mar 22, 2022

Oh wow … 🥳

The only thing is that the _dispose() is deliberately staged for a call from the event loop, because __del__ is not called from the event loop, and depending on what _dispose() does, this might be a problem. I’ll have a look.

almarklein on Apr 12, 2022

Here is a video of slow opening/closing of the app with gc.collect() added at the top of init

https://user-images.githubusercontent.com/10289651/162928781-2bbeef61-073a-45e2-8068-aa9caef6dee6.mp4

matkuki on Apr 12, 2022

I meant that it looked suspicious, because what happens there is a bit weird (it actually prevents the object being collected by creating a new reference to it). But in retrospect I don’t think it should matter for this case; when the session closes, it first calls dispose on the root component, and then releases the reference to it, so that __del__ should be a no-op.

I just tried on Windows, and I can see that the memory does not increase if I use the extra gc.collect() call. However, if I refresh in quick succession, it does increase. What happens then, is that new connections are made, but since the old ones take a second or so to disconnect, multiple sessions are active at a time. After a minute or so, if I then refresh a few times (at a gentle pace), I see the memory return to normal again (looks like it need multipe calls to gc.collect() to do a complete collection.

Another thing I noticed, is that if I refresh gently, I see the dispose being called. But if I refresh quickly, I don’t. It looks like the session exists so short, that the application is created, but not registered yet with the session, so it is not cleaned up correctly …

almarklein on Apr 12, 2022

I have just tried adding gc.collect() at the top of init(), and indeed the __del__ gets called after the second connection, but memory usage is still going up as before, no changes. After six reloads the memory is at about 3GB, which is huge. Note that this is on Windows 10.

matkuki on Apr 11, 2022

@almarklein No problem, here you go. Just keep opening and closing http://localhost:49190/demo/ in a browser tab and the memory should jump by about 500MB each time, and __del__ never gets called (it only gets called for all widget instances when you CTRL+C the app). Note that this is on Windows, haven’t tested on Linux.

from flexx import flx

class Demo(flx.PyWidget):
    
    def __del__(self):
        print("Demo-__del__")
        super().__del__()
    
    def dispose(self):
        print("Demo-dispose")
        super().dispose()
    
    def init(self):
        super().init()
        
        self.big_list_0 = []
        for i in range(100_000):
            new_item = {}
            for j in range(100):
                new_item[j] = "Some test string things"
            self.big_list_0.append(new_item)
        print("big_list_0 created")
        
        with flx.PinboardLayout(flex=1) as self.frame:
            self.label = flx.Label(text="Test", flex=0, style="color: black; font-size: 30pt; width: 200px; height: 200px;")
        


if __name__ == '__main__':
    a = flx.App(Demo, title='Flexx demo')
    a.serve("demo")
    flx.start()

matkuki on Mar 22, 2022