ipython: Autoreload slow

On upgrading from 7.5 to 7.6.1, autoreload in Jupyter Notebook is much slower, to the point of seeming to hang for a >minute. When I lose my patience and interrupt execution, I get a traceback like this

[autoreload of algos.ppo.nets failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 245, in check
    superreload(m, reload, self.old_objects)
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 450, in superreload
    update_generic(old_obj, new_obj)
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 387, in update_generic
    update(a, b)
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 357, in update_class
    update_instances(old, new)
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 317, in update_instances
    update_instances(old, new, obj, visited)
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 317, in update_instances
    update_instances(old, new, obj, visited)
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 317, in update_instances
    update_instances(old, new, obj, visited)
  [Previous line repeated 2 more times]
  File "/opt/conda/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 300, in update_instances
    for obj in (obj for obj in objects if id(obj) not in visited):
KeyboardInterrupt
]

I don’t have a minimal example I’m afraid, but if anyone arrives here Googling for a similar problem, rolling back to 7.5 fixes it.

My specs are

Ubuntu 18.04.2 LTS Python 3.73 Jupyter:

jupyter==1.0.0
jupyter-client==5.3.0
jupyter-console==6.0.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.5.0
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-nbextensions-configurator==0.4.1

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 9
Comments: 19 (10 by maintainers)

Commits related to this issue

Try to speedup autoreload by walking all the object using gc. Unsure if this work; we were avoid this initially as this crawl more objects; but at the same time we do less objects introspection. Ano... — committed to Carreau/ipython by Carreau 5 years ago

Most upvoted comments

I can’t add further debug info, but I had a similar problem with ipython-7.6.1 and Spyder-3.3.6:

open ipython console
select some code in the editor
hit F9 (this essentially copies the code to the console and executes it)

Every other time, this causes ipython to hang until interrupted by ctr+c, which then gives a stacktrace as pasted above. Repeating the exact same procedure works fine on the next try. Downgrading to ipython-7.5 resolved the issues.

This is all on a Ubuntu 16.04, with Spyder and ipython installed via pip3:

pip3 install --user --upgrade spyder3

To fix the issue:

pip3 install --user ipython==7.5

Is there further info I could provide?

smerzbach on Jul 19, 2019

This problem seems to be extra bad when I’m working with large pandas DataFrames; is it possible that every single string value is getting scanned by the autoreload logic? If so is there any possible way to work around this case?

bnaul on Aug 5, 2019

Same problem.

Add a few extra info:

The larger the RAM cost of the session is, the longer the hanging lasts.
If you wait until the hanging finishes, the reload actually worked fine. All instances of classes were reload as expected.
If you stop the hanging in the middle, again the reload actually worked, and will still work in future reload in this IPython session.
If you stop the hanging real fast like right after you execute the cell, the autoreload won’t work this time and won’t work in this IPython session.

WestXu on Jul 25, 2019

In the attached session, changing any of the imported modules causes autoreload to hang: https://drive.google.com/file/d/1auE2ssFabG6Pgwl7jZPA2yRTqg6-xUcV/view?usp=sharing

Since I guess this dump contains much of my source code (due to nested imports and such) I’m going to have to share privately… so you need to use a google drive account and lmk when you request access so I’ll immediately approve… same goes for anyone else who has time to look at it.

Sent you a request (Niclas)

Naikless on Aug 7, 2019