scipy: scipy.io.savemat does not handle None properly

The MATLAB equivalent of None is the empty array []. It looks like scipy does not perform this conversion.

But if I try to save a dict with a None inside it, I get this: (scipy 0.13.0 in Anaconda Python on Windows 7 64-bit)

>>> import scipy.io
>>> scipy.io.savemat('test.mat',{'x':3})
>>> scipy.io.savemat('test.mat',{'x':None})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\app\python\anaconda\1.6.0\envs\daqlogger\lib\site-packages\scipy\io\m
atlab\mio.py", line 204, in savemat
    MW.put_variables(mdict)
  File "c:\app\python\anaconda\1.6.0\envs\daqlogger\lib\site-packages\scipy\io\m
atlab\mio5.py", line 872, in put_variables
    self._matrix_writer.write_top(var, asbytes(name), is_global)
  File "c:\app\python\anaconda\1.6.0\envs\daqlogger\lib\site-packages\scipy\io\m
atlab\mio5.py", line 622, in write_top
    self.write(arr)
  File "c:\app\python\anaconda\1.6.0\envs\daqlogger\lib\site-packages\scipy\io\m
atlab\mio5.py", line 643, in write
    % (arr, type(arr)))
TypeError: Could not convert None (type <type 'NoneType'>) to array

About this issue

Original URL
State: open
Created 10 years ago
Reactions: 1
Comments: 25 (10 by maintainers)

Most upvoted comments

I would suggest:

scipy.io.savemat(..., none_object=())

and that none_object=None by default for backwards compatibility.

pbrod on Nov 24, 2017

Definitely could use a solution to this.

randerson57261 on Apr 28, 2021

I’m also running into this issue, and my use case is very similar to jason-s. I’ve hacked together a solution when I’m populating my nested structures that replaces python None with NaN, because that is what makes the most sense for my use case.

According to google there are tools for actual json io with matlab. Is it really better to read the json into python and then write it using a function that was designed to save matrices?

I turned to doing this processing with scipy because I can trivially process large files line-by-line in python. Maybe processing line-by-line in MATLAB is doable, but I wasn’t able to get it working without prior knowledge of what the objects look like or with any sort of reasonable speed. When my log files look something like . . .

{"json-object-1": {potentially large sub object goes here},
   "date": 20181121,
    "time":  090000.0}
{json-object-2-with-timestamp}
{same json-object-1, different time stamp}
{json-object-3}
{you get the idea, it's all sorts of different json-objects concatenated into the same file}

Processing line by line is, by design, what I’d like to be doing. I’ve had great success using scipy.io.savemat, but adding this feature would have saved a decent amount of development time.

I’d like to contribute to this fix myself and save the next fellow the trouble if a decision is made!

CJSmith-0141 on Nov 21, 2018