astropy: CSV table writer fails with cryptic message when meta contains "comments" key

I ran into this cryptic TypeError (with current Astropy, 3.1.dev21648)

>>> from astropy.table import Table
>>> table = Table.read('https://www.mpi-hd.mpg.de/hfm/HESS/hgps/data/hgps_catalog_v1.fits.gz', hdu=1)
Downloading https://www.mpi-hd.mpg.de/hfm/HESS/hgps/data/hgps_catalog_v1.fits.gz
|=====================================================================================================================|  49k/ 49k (100.00%)         0s
>>> table.write('test.csv', format='ascii.csv', overwrite=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/table/table.py", line 2560, in write
    io_registry.write(self, *args, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/registry.py", line 560, in write
    writer(data, *args, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/connect.py", line 43, in io_write
    return write(table, filename, format=format, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/ui.py", line 880, in write
    writer.write(table, output)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
    self._write(table, output, {'fill_values': [(core.masked, '')]})
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 171, in _write
    writer.write(output, header_output, output_types)
  File "astropy/io/ascii/cparser.pyx", line 1024, in astropy.io.ascii.cparser.FastWriter.write
  File "astropy/io/ascii/cparser.pyx", line 1004, in astropy.io.ascii.cparser.FastWriter._write_header
  File "astropy/io/ascii/cparser.pyx", line 995, in astropy.io.ascii.cparser.FastWriter._write_comments
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

I think the error is a bug in the CSV writer when meta contains a key named comments. Here’s a minimal example to trigger similar errors:

>>> table = Table(data={'a': [1, 2]}, meta={'comments': 'spam'})
>>> table
<Table length=2>
  a  
int64
-----
    1
    2
>>> table.meta
{'comments': 'spam'}
>>> table.write('test.csv', format='ascii.csv', overwrite=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/table/table.py", line 2560, in write
    io_registry.write(self, *args, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/registry.py", line 560, in write
    writer(data, *args, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/connect.py", line 43, in io_write
    return write(table, filename, format=format, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/ui.py", line 880, in write
    writer.write(table, output)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
    self._write(table, output, {'fill_values': [(core.masked, '')]})
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 170, in _write
    writer = cparser.FastWriter(table, **write_kwargs)
  File "astropy/io/ascii/cparser.pyx", line 966, in astropy.io.ascii.cparser.FastWriter.__cinit__
TypeError: Expected list, got str

That error is better, but still doesn’t mention that the problem is related to the comments key in table.meta.

Let’s try putting a list like the TypeError message suggested:

>>> table = Table(data={'a': [1, 2]}, meta={'comments': ['spam']})
>>> table.write('test.csv', format='ascii.csv', overwrite=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/table/table.py", line 2560, in write
    io_registry.write(self, *args, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/registry.py", line 560, in write
    writer(data, *args, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/connect.py", line 43, in io_write
    return write(table, filename, format=format, **kwargs)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/ui.py", line 880, in write
    writer.write(table, output)
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 194, in write
    self._write(table, output, {'fill_values': [(core.masked, '')]})
  File "/Users/deil/Library/Python/3.6/lib/python/site-packages/astropy/io/ascii/fastbasic.py", line 171, in _write
    writer.write(output, header_output, output_types)
  File "astropy/io/ascii/cparser.pyx", line 1024, in astropy.io.ascii.cparser.FastWriter.write
  File "astropy/io/ascii/cparser.pyx", line 1004, in astropy.io.ascii.cparser.FastWriter._write_header
  File "astropy/io/ascii/cparser.pyx", line 995, in astropy.io.ascii.cparser.FastWriter._write_comments
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Now we get the

TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

that I got from my actual FITS file that had COMMENTS in the header.

@taldcroft or anyone working on table these days - can this be fixed or the error messages improved?

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 2
  • Comments: 18 (18 by maintainers)

Most upvoted comments

I am procrastinating, so I’ll have a look.

FWIW, a work-around is to specify the comment option:

>>> table = Table(data={'a': [1, 2]}, meta={'comments': ['spam']})
>>> table.write('test.csv', format='ascii.csv', overwrite=True, comment='#')

This works with both the fast C writer and the pure Python writer.

The root cause is at https://github.com/astropy/astropy/blob/732d89c2940156bdc0e200bb36dc38b5e424bcba/astropy/io/ascii/basic.py#L234 and here https://github.com/astropy/astropy/blob/732d89c2940156bdc0e200bb36dc38b5e424bcba/astropy/io/ascii/fastbasic.py#L187

I think the original motivation for this is that a pure CSV file cannot have comments. See https://tools.ietf.org/html/rfc4180 and https://en.wikipedia.org/wiki/Comma-separated_values. But of course the user can overwrite from the write or read function calls, if they know to do so.

So I think the answer here is better error handling and messages, and maybe some documentation.