xarray: holoviews / bokeh doesn't like cftime coords

Code Sample, a copy-pastable example if possible

Consider a simple working example of converting an xarray dataset to holoviews for plotting:

ref_date = '1981-01-01'
ds = xr.DataArray([1, 2, 3], dims=['time'],
                  coords={'time': ('time', [1, 2, 3],
                                   {'units': 'days since %s' % ref_date})}
                  ).to_dataset(name='foo')
with xr.set_options(enable_cftimeindex=True):
    ds = xr.decode_cf(ds)
print(ds)
hv_ds = hv.Dataset(ds)
hv_ds.to(hv.Curve)

This gives

<xarray.Dataset>
Dimensions:  (time: 3)
Coordinates:
  * time     (time) datetime64[ns] 1981-01-02 1981-01-03 1981-01-04
Data variables:
    foo      (time) int64 ...

and

Problem description

Now change ref_date = '0181-01-01' (or anything outside of the valid range for regular pandas datetime index). We get a beautiful new cftimeindex

<xarray.Dataset>
Dimensions:  (time: 3)
Coordinates:
  * time     (time) object 0181-01-02 00:00:00 0181-01-03 00:00:00 ...
Data variables:
    foo      (time) int64 ...

but holoviews / bokeh doesn’t like it

/opt/conda/lib/python3.6/site-packages/xarray/coding/times.py:132: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy cftime.datetime objects instead, reason: dates out of range
  enable_cftimeindex)
/opt/conda/lib/python3.6/site-packages/xarray/coding/variables.py:66: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy cftime.datetime objects instead, reason: dates out of range
  return self.func(self.array[key])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj, include, exclude)
    968 
    969             if method is not None:
--> 970                 return method(include=include, exclude=exclude)
    971             return None
    972         else:

/opt/conda/lib/python3.6/site-packages/holoviews/core/dimension.py in _repr_mimebundle_(self, include, exclude)
   1229         combined and returned.
   1230         """
-> 1231         return Store.render(self)
   1232 
   1233 

/opt/conda/lib/python3.6/site-packages/holoviews/core/options.py in render(cls, obj)
   1287         data, metadata = {}, {}
   1288         for hook in hooks:
-> 1289             ret = hook(obj)
   1290             if ret is None:
   1291                 continue

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in pprint_display(obj)
    278     if not ip.display_formatter.formatters['text/plain'].pprint:
    279         return None
--> 280     return display(obj, raw_output=True)
    281 
    282 

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in display(obj, raw_output, **kwargs)
    248     elif isinstance(obj, (CompositeOverlay, ViewableElement)):
    249         with option_state(obj):
--> 250             output = element_display(obj)
    251     elif isinstance(obj, (Layout, NdLayout, AdjointLayout)):
    252         with option_state(obj):

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in wrapped(element)
    140         try:
    141             max_frames = OutputSettings.options['max_frames']
--> 142             mimebundle = fn(element, max_frames=max_frames)
    143             if mimebundle is None:
    144                 return {}, {}

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in element_display(element, max_frames)
    186         return None
    187 
--> 188     return render(element)
    189 
    190 

/opt/conda/lib/python3.6/site-packages/holoviews/ipython/display_hooks.py in render(obj, **kwargs)
     63         renderer = renderer.instance(fig='png')
     64 
---> 65     return renderer.components(obj, **kwargs)
     66 
     67 

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/renderer.py in components(self, obj, fmt, comm, **kwargs)
    257         # Bokeh has to handle comms directly in <0.12.15
    258         comm = False if bokeh_version < '0.12.15' else comm
--> 259         return super(BokehRenderer, self).components(obj,fmt, comm, **kwargs)
    260 
    261 

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/renderer.py in components(self, obj, fmt, comm, **kwargs)
    319             plot = obj
    320         else:
--> 321             plot, fmt = self._validate(obj, fmt)
    322 
    323         widget_id = None

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/renderer.py in _validate(self, obj, fmt, **kwargs)
    218         if isinstance(obj, tuple(self.widgets.values())):
    219             return obj, 'html'
--> 220         plot = self.get_plot(obj, renderer=self, **kwargs)
    221 
    222         fig_formats = self.mode_formats['fig'][self.mode]

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/renderer.py in get_plot(self_or_cls, obj, doc, renderer)
    150             doc = Document() if self_or_cls.notebook_context else curdoc()
    151         doc.theme = self_or_cls.theme
--> 152         plot = super(BokehRenderer, self_or_cls).get_plot(obj, renderer)
    153         plot.document = doc
    154         return plot

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/renderer.py in get_plot(self_or_cls, obj, renderer)
    205             init_key = tuple(v if d is None else d for v, d in
    206                              zip(plot.keys[0], defaults))
--> 207             plot.update(init_key)
    208         else:
    209             plot = obj

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/plot.py in update(self, key)
    511     def update(self, key):
    512         if len(self) == 1 and ((key == 0) or (key == self.keys[0])) and not self.drawn:
--> 513             return self.initialize_plot()
    514         item = self.__getitem__(key)
    515         self.traverse(lambda x: setattr(x, '_updated', True))

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py in initialize_plot(self, ranges, plot, plots, source)
    729         if not self.overlaid:
    730             self._update_plot(key, plot, style_element)
--> 731             self._update_ranges(style_element, ranges)
    732 
    733         for cb in self.callbacks:

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py in _update_ranges(self, element, ranges)
    498         if not self.drawn or xupdate:
    499             self._update_range(x_range, l, r, xfactors, self.invert_xaxis,
--> 500                                self._shared['x'], self.logx, streaming)
    501         if not self.drawn or yupdate:
    502             self._update_range(y_range, b, t, yfactors, self.invert_yaxis,

/opt/conda/lib/python3.6/site-packages/holoviews/plotting/bokeh/element.py in _update_range(self, axis_range, low, high, factors, invert, shared, log, streaming)
    525             updates = {}
    526             if low is not None and (isinstance(low, util.datetime_types)
--> 527                                     or np.isfinite(low)):
    528                 updates['start'] = (axis_range.start, low)
    529             if high is not None and (isinstance(high, util.datetime_types)

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Similar but slightly different errors arise for different holoviews types (e.g. hv.Image) and contexts (using time as a holoviews kdim).

Expected Output

This should work.

I’m not sure if this is really an xarray problem. Maybe it needs a fix in holoviews (or bokeh). But I’m raising it here first since clearly we have introduced this new wrinkle in the stack. Cc’ing @philippjfr since he is the expert on all things holoviews.

Output of `xr.show_versions()`

INSTALLED VERSIONS ------------------ commit: None python: 3.6.3.final.0 python-bits: 64 OS: Linux OS-release: 4.4.111+ machine: x86_64 processor: x86_64 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

xarray: 0.10.4 pandas: 0.23.0 numpy: 1.14.3 scipy: 1.1.0 netCDF4: 1.4.0 h5netcdf: None h5py: None Nio: None zarr: 2.2.0 bottleneck: None cyordereddict: None dask: 0.17.5 distributed: 1.21.8 matplotlib: 2.2.2 cartopy: None seaborn: None setuptools: 39.0.1 pip: 10.0.1 conda: 4.3.34 pytest: 3.5.1 IPython: 6.3.1 sphinx: None

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 16 (14 by maintainers)

Most upvoted comments

@jbusecke I recently stumbled upon https://github.com/SciTools/nc-time-axis. You can install it through conda-forge:

$ conda install -c conda-forge nc-time-axis

However, for now you won’t be able to use xarray’s built in plotting, since xarray will raise an error if you try to plot anything with coordinates that aren’t numeric or of type datetime.datetime or np.datetime64 (but nc-time-axis at least gives you matplotlib support). You’ll also need to convert your dates to nc_time_axis.CalendarDateTime objects; you can do that with a simple list comprehension (see the example in their README).

spencerkclark on Jul 11, 2018

From a cursory examination, it sure looks (to me) that nc_time_axis could be made to directly support plotting of cftime.datetime classes. This could probably either be done in a separate package or upstream in cftime.

shoyer on Jul 13, 2018

Thanks for your follow up comment. I understand much better where you are coming from now! We do sincerely appreciate your feedback and contributions. Yes: we definitely want xarray to be adopted widely!

There is no doubt that these weird calendars are a continuous source of frustration. The challenge is compounded by the fast that, basically, only the climate community needs them. All of the fancy time indexing stuff in pandas is most likely there because of its value to the finance community.

Maybe a good path forward would be for @spencerkclark to outline what steps might be needed to get xarray’s built in plotting to work with cftimeindex. That way, anyone who urgently needs this feature has some sort of roadmap for starting to implement it themselves.

rabernat on Jul 12, 2018

Darn. Just when I thought the time stuff was sorted. This is (yet another) deal breaker as far as recommending mass adoption goes.

@aidanheerdegen – support for unconventional time coordinates (via cftime) is not a trivial problem–there are many special cases and complex logic to deal with. Rather than being discouraged by this bug, I encourage you to take a longer view to see how much this support has improved over the past year. One year ago, xarray could not have decoded this time coordinate at all!

I’m curious what you are referring to with your comment about “mass adoption.” To whom are you making recommendations about adoption of xarray? And what do you consider the numerous other dealbreakers to be? I hope you see that these blanket comments could be read as quite discouraging and negative to the volunteer developers working hard on xarray.

Is there an estimate when, or if, cftime indexes will be supported by xarray’s .plot() method?

The work on cftimeindex is being done entirely on a volunteer basis by graduate students like @spencerkclark (👏 👏 👏). In order to make xarray development progress faster, we need more contributors to the project. In my view, the ideal place to recruit such contributors is from within the paid computational support staff from major climate modeling centers. These people are ideally poised to understand what features are most important to users, and they are actually paid to help develop tools (like xarray) to help users be more productive. We recently created a contributor guide to make it easier for new contributors to come on board. If you can think of anyone who might be interested in contributing to xarray, please let us know, either here or via a private channel.