ipython: autoreload magic doesn't reload objects

update: the problem appears to be with any object, unrelated to pickle. pickle was just exposing it in a forceful way. Skip to the latest diagnostic https://github.com/ipython/ipython/issues/11588#issuecomment-464949800

when autoreload reloads a python module, it appears to not be fully reloading the already instantiated objects, and pickle fails to pickle them with:

Can't pickle <class 'mytestclass.Test'>: it's not the same object as mytestclass.Test

Here is a notebook to reproduce, must be run in 3 parts (or 3 cells):

# cell1
%reload_ext autoreload
%autoreload 2

import pickle

file = "mytestclass.py"

text = """
class Test():
    def __init__(self):
        self.ok = 1
"""
with open("mytestclass.py","w") as f: f.write(text)

from mytestclass import *
test = Test()

# works
p = pickle.dumps(test)

# cell2
# touch forces autoreloader to reload the library file
!echo touch $file
!touch $file

# cell3
# now it fails
p = pickle.dumps(test)

The error is:

In [6]: p = pickle.dumps(test)
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-6-740c326b650e> in <module>()
----> 1 p = pickle.dumps(test)

PicklingError: Can't pickle <class 'mytestclass.Test'>: it's not the same object as mytestclass.Test

Please note that it should be run in 3 parts - if it’s run in one go the reload effect won’t be activated. Can be run in jupyter notebook or directly in ipython shell.

I traced why pickle fails and expanded on it in this issue https://github.com/fastai/fastai/issues/1493 - tldr; it compares memory addresses of the object’s class and the class in sys.modules - if their memory addresses don’t match it refuses to pickle.

I think there are other issues with the incomplete reload, (i.e. object methods disappearing after reload), but I’m yet to make reproducible cases for those. I believe pickle is just exposing a deeper problem.

I tried the git master ipython and went all the way back to ipython-6.0.0 - the problem is the same.

And yes, I read the caveats section, that does say that reload isn’t 100% perfect.

Thank you.