astropy: CSV table writer fails with cryptic message when meta contains "comments" key
I ran into this cryptic TypeError (with current Astropy, 3.1.dev21648
)
>>> from astropy.table import Table
>>> table = Table.read('https://www.mpi-hd.mpg.de/hfm/HESS/hgps/data/hgps_catalog_v1.fits.gz', hdu=1)
Downloading https://www.mpi-hd.mpg.de/hfm/HESS/hgps/data/hgps_catalog_v1.fits.gz
|=====================================================================================================================| 49k/ 49k (100.00%) 0s
>>> table.write('test.csv', format='ascii.csv', overwrite=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/table/table.py", line 2560, in write
io_registry.write(self, *args, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/registry.py", line 560, in write
writer(data, *args, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/connect.py", line 43, in io_write
return write(table, filename, format=format, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/ui.py", line 880, in write
writer.write(table, output)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
self._write(table, output, {'fill_values': [(core.masked, '')]})
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 171, in _write
writer.write(output, header_output, output_types)
File "astropy/io/ascii/cparser.pyx", line 1024, in astropy.io.ascii.cparser.FastWriter.write
File "astropy/io/ascii/cparser.pyx", line 1004, in astropy.io.ascii.cparser.FastWriter._write_header
File "astropy/io/ascii/cparser.pyx", line 995, in astropy.io.ascii.cparser.FastWriter._write_comments
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
I think the error is a bug in the CSV writer when meta
contains a key named comments
.
Here’s a minimal example to trigger similar errors:
>>> table = Table(data={'a': [1, 2]}, meta={'comments': 'spam'})
>>> table
<Table length=2>
a
int64
-----
1
2
>>> table.meta
{'comments': 'spam'}
>>> table.write('test.csv', format='ascii.csv', overwrite=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/table/table.py", line 2560, in write
io_registry.write(self, *args, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/registry.py", line 560, in write
writer(data, *args, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/connect.py", line 43, in io_write
return write(table, filename, format=format, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/ui.py", line 880, in write
writer.write(table, output)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
self._write(table, output, {'fill_values': [(core.masked, '')]})
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 170, in _write
writer = cparser.FastWriter(table, **write_kwargs)
File "astropy/io/ascii/cparser.pyx", line 966, in astropy.io.ascii.cparser.FastWriter.__cinit__
TypeError: Expected list, got str
That error is better, but still doesn’t mention that the problem is related to the comments
key in table.meta
.
Let’s try putting a list like the TypeError message suggested:
>>> table = Table(data={'a': [1, 2]}, meta={'comments': ['spam']})
>>> table.write('test.csv', format='ascii.csv', overwrite=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/table/table.py", line 2560, in write
io_registry.write(self, *args, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/registry.py", line 560, in write
writer(data, *args, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/connect.py", line 43, in io_write
return write(table, filename, format=format, **kwargs)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/ui.py", line 880, in write
writer.write(table, output)
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
self._write(table, output, {'fill_values': [(core.masked, '')]})
File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 171, in _write
writer.write(output, header_output, output_types)
File "astropy/io/ascii/cparser.pyx", line 1024, in astropy.io.ascii.cparser.FastWriter.write
File "astropy/io/ascii/cparser.pyx", line 1004, in astropy.io.ascii.cparser.FastWriter._write_header
File "astropy/io/ascii/cparser.pyx", line 995, in astropy.io.ascii.cparser.FastWriter._write_comments
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
Now we get the
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
that I got from my actual FITS file that had COMMENTS
in the header.
@taldcroft or anyone working on table these days - can this be fixed or the error messages improved?
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 18 (18 by maintainers)
I am procrastinating, so I’ll have a look.
FWIW, a work-around is to specify the
comment
option:This works with both the fast C writer and the pure Python writer.
The root cause is at https://github.com/astropy/astropy/blob/732d89c2940156bdc0e200bb36dc38b5e424bcba/astropy/io/ascii/basic.py#L234 and here https://github.com/astropy/astropy/blob/732d89c2940156bdc0e200bb36dc38b5e424bcba/astropy/io/ascii/fastbasic.py#L187
I think the original motivation for this is that a pure CSV file cannot have comments. See https://tools.ietf.org/html/rfc4180 and https://en.wikipedia.org/wiki/Comma-separated_values. But of course the user can overwrite from the
write
orread
function calls, if they know to do so.So I think the answer here is better error handling and messages, and maybe some documentation.