hvplot: hvplot pickle incompatible

Describe the bug

I have a collection of holoviews and hvplot plots that I’d like to pickle and store for later retrieval, ideally including all styles/customizations.

With holoviews, this can easily be done with using either pickle or hv.Store - where the first stores the core holoviews obj and the latter includes the holoviews options to persist style/customizations.

However, neither pickle or hv.Store seem to work with hvplot plots.

Steps/Code to reproduce bug

Create a hvplot:

import numpy as np
import pandas as pd
import hvplot.pandas

df = pd.DataFrame(np.random.randn(1000, 4), columns=list('ABCD')).cumsum()
hv_plot_example = df.hvplot()
hv_plot_example

Try to pickle the plot:

import pickle

plot_bytes = pickle.dumps(hv_plot_example)

import holoviews as hv

plot_bytes = hv.Store.dumps(hv_plot_example)

Both throw the following AttributeError:

AttributeError: Can't pickle local object Redim._create_expression_transform.<locals>._transform_expression

Expected behavior

I would like to be able to pickle the hvplot directly so that I can persist in for retrieval later. Without this functionality, I’m limited to writing plots to temporary .png files and storing those rather than the pickled object.

Additional Context

holoviews supports this functionality as the following example works perfectly:

import holoviews as hv
hv.extension('bokeh')

holo_plot_example = hv.Overlay()
for column in list(df):
    ys = df[column].values
    holo_plot_example *= hv.Curve(ys).opts(title="Test")
    
holo_plot_example

which can be pickled with either:

import pickle

holo_plot_pickle = pickle.dumps(holo_plot_example)
pickle.loads(holo_plot_pickle)

holo_plot_store_pickle = hv.Store.dumps(holo_plot_example)
hv.Store.loads(holo_plot_store_pickle)

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 15 (5 by maintainers)

Most upvoted comments

Confirmed that this is fixed with hvplot 0.7.0 and holoviews 1.14.1. Thanks @jlstevens for the heads up. I’ll close out the issue.

joe-wolfe21 on Jan 13, 2021

Some recent changes to holoviews might have fixed this. I remember we added some code to clear out unserializable callbacks from .pipeline iirc so maybe this works now. Worth reviewing.

jlstevens on Jan 13, 2021

… it’s a HoloViews issue and it will require a significant amount of effort to make HoloViews objects fully picklable at this point

The key word for me here is fully. I would expect HoloViews objects to still be pickleable unless the user introduces things that clearly aren’t pickleable (e.g lambdas). If there are exceptions I would like to know why and list them explicitly in the docs.

jlstevens on Nov 4, 2020

I also experience this bug with HoloViews 1.13.4 and hvplot 0.6.0.

I have started to use DiskCache and it uses pickle. But it cannot pickle hvplot like in the example below.

import time

import hvplot.pandas
import numpy as np
import pandas as pd
import panel as pn
import param

from diskcache import FanoutCache

EXPIRE = 5 * 60  # 5 minutes
CACHE_DIRECTORY = "cache"

# cache=pn.state.cache # Simple builtin dictionary cache
cache=FanoutCache(directory=CACHE_DIRECTORY)

@cache.memoize()
def _get_data(value):
    print("loading data", value)
    time.sleep(1)
    print("data loaded")
    return pd.DataFrame(np.random.randint(0, 100, size=(100, 4)), columns=list("ABCD"))

@cache.memoize()
def _get_plot(dataframe):
    return dataframe.hvplot(x="A", y="B")


class MyApp(param.Parameterized):
    value = param.Integer(default=0, bounds=(0, 100))

    data = param.DataFrame()
    plot = param.Parameter()
    view = param.Parameter()

    def __init__(self, **params):
        super().__init__(**params)

        self.plot_panel = pn.pane.HoloViews()

        self.view = pn.Column(self.param.value, self.plot_panel)

        self._update_data()

    @param.depends("value", watch=True)
    def _update_data(self):
        self.data = _get_data(self.value)

    @param.depends("data", watch=True)
    def _update_plot(self):
        self.plot = _get_plot(self.data)

    @param.depends("plot", watch=True)
    def _update_plot_panel(self):
        self.plot_panel.object = self.plot


if __name__.startswith("bokeh"):
    pn.config.sizing_mode = "stretch_width"
    MyApp().view.servable()

$ python -m panel serve 'scripts\issue_hvplot_pickle.py' --dev
2020-11-03 07:32:04,233 Starting Bokeh server version 2.2.3 (running on Tornado 6.0.4)
2020-11-03 07:32:04,235 User authentication hooks NOT provided (default user enabled)
2020-11-03 07:32:04,238 Bokeh app running at: http://localhost:5006/issue_hvplot_pickle
2020-11-03 07:32:04,238 Starting Bokeh server with process id: 24252
loading data 0
data loaded
2020-11-03 07:32:11,923 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x00000281589C85C8>: Can't pickle local object 'Redim._create_expression_transform.<locals>._transform_expression'
File "core.py", line 234, in store:
result = pickle.dumps(value, protocol=self.pickle_protocol) Traceback (most recent call last):
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\bokeh\application\handlers\code_runner.py", line 197, in run
    exec(self._code, module.__dict__)
  File "C:\repos\private\awesome-panel\scripts\issue_hvplot_pickle.py", line 60, in <module>
    MyApp().view.servable()
  File "C:\repos\private\awesome-panel\scripts\issue_hvplot_pickle.py", line 43, in __init__
    self._update_data()
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\param\parameterized.py", line 337, in _depends
    return func(*args,**kw)
  File "C:\repos\private\awesome-panel\scripts\issue_hvplot_pickle.py", line 47, in _update_data
    self.data = _get_data(self.value)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\param\parameterized.py", line 304, in _f
    return f(self, obj, val)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\param\parameterized.py", line 915, in __set__
    obj.param._call_watcher(watcher, event)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\param\parameterized.py", line 1554, in _call_watcher
    watcher.fn(event)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\param\parameterized.py", line 499, in caller
    return getattr(self,n)()
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\param\parameterized.py", line 337, in _depends
    return func(*args,**kw)
  File "C:\repos\private\awesome-panel\scripts\issue_hvplot_pickle.py", line 51, in _update_plot
    self.plot = _get_plot(self.data)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\diskcache\core.py", line 1861, in wrapper
    self.set(key, result, expire, tag=tag, retry=True)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\diskcache\fanout.py", line 86, in set
    return shard.set(key, value, expire, read, tag, retry)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\diskcache\core.py", line 772, in set
    size, mode, filename, db_value = self._disk.store(value, read, key=key)
  File "C:\repos\private\awesome-panel\.venv\lib\site-packages\diskcache\core.py", line 234, in store
    result = pickle.dumps(value, protocol=self.pickle_protocol)
AttributeError: Can't pickle local object 'Redim._create_expression_transform.<locals>._transform_expression'

2020-11-03 07:32:12,044 WebSocket connection opened
2020-11-03 07:32:12,045 ServerConnection created

MarcSkovMadsen on Nov 3, 2020

@philippjfr or @jlstevens would know which of those two is more appropriate here. It doesn’t look terribly difficult to make it picklable; I think we’d just need to define a named class that accepts kdims, vidms, and exclude lists as instance attributes and has the code for _transform_expression as self.__call__(); the resulting object should be fully picklable. But there may be other similar usages that would need the same treatment, so I’ll leave it to the others to weigh in.

jbednar on Sep 30, 2020

Good question! I would guess that there’s a way to reproduce this with HoloViews alone, because the local object in question is defined in HoloViews (https://github.com/holoviz/holoviews/blob/master/holoviews/core/accessors.py#L370). It must somehow be used automatically by hvPlot, while requiring some explicit invocation in HoloViews.

In any case, it seems like a reasonable request, i.e. either to make the _transform_expression object picklable or to have hvPlot avoid creating it or storing it by default.

jbednar on Sep 30, 2020