astropy: FITS tables with a column “a: 0” produce duplicate TTYPE entries

Description

astropy.table fails to save correctly FITS tables containing a column named “a: 0” and cannot load the generated file. This occurs when the column name starts with a letter, then contains a colon, then a space, then a number. In this case the TTYPE entry appears to be duplicated. For instance with the above example the FITS header generated by astropy.table is the following:

XTENSION= ‘BINTABLE’ / binary table extension
BITPIX = 8 / array data type
NAXIS = 2 / number of array dimensions
NAXIS1 = 8 / length of dimension 1
NAXIS2 = 1 / length of dimension 2
PCOUNT = 0 / number of group parameters
GCOUNT = 1 / number of groups
TFIELDS = 1 / number of table fields
TTYPE1 = ‘a: 0’
TTYPE1 = ‘a: 0’
TFORM1 = 'D ’
TTYPE1 = ‘a: 0’

Attempting to read this file with astropy.table leads to a ValueError exception:

ValueError                                Traceback (most recent call last)
<ipython-input-2-b76ff7c17fe6> in <module>
----> 1 t = Table.read('8d4584eb-4f4b-4215-b138-5070614b49f7.fits')

/usr/lib/python3.8/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
     50     def __call__(self, *args, **kwargs):
     51         cls = self._cls
---> 52         out = registry.read(cls, *args, **kwargs)
     53 
     54         # For some readers (e.g., ascii.ecsv), the returned `out` class is not

/usr/lib/python3.8/site-packages/astropy/io/registry.py in read(cls, format, *args, **kwargs)
    521 
    522         reader = get_reader(format, cls)
--> 523         data = reader(*args, **kwargs)
    524 
    525         if not isinstance(data, cls):

/usr/lib/python3.8/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
    213 
    214         try:
--> 215             return read_table_fits(hdulist, hdu=hdu,
    216                                    astropy_native=astropy_native)
    217         finally:

/usr/lib/python3.8/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
    226     # In the loop below we access the data using data[col.name] rather than
    227     # col.array to make sure that the data is scaled correctly if needed.
--> 228     data = table.data
    229 
    230     columns = []

/usr/lib/python3.8/site-packages/astropy/utils/decorators.py in __get__(self, obj, owner)
    756                 return val
    757             else:
--> 758                 val = self.fget(obj)
    759                 obj.__dict__[self._key] = val
    760                 return val

/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py in data(self)
    397     @lazyproperty
    398     def data(self):
--> 399         data = self._get_tbdata()
    400         data._coldefs = self.columns
    401         data._character_as_bytes = self._character_as_bytes

/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py in _get_tbdata(self)
    169                                                type=np.rec.recarray)
    170         else:
--> 171             raw_data = self._get_raw_data(self._nrows, columns.dtype,
    172                                           self._data_offset)
    173             if raw_data is None:

/usr/lib/python3.8/site-packages/astropy/utils/decorators.py in __get__(self, obj, owner)
    756                 return val
    757             else:
--> 758                 val = self.fget(obj)
    759                 obj.__dict__[self._key] = val
    760                 return val

/usr/lib/python3.8/site-packages/astropy/io/fits/column.py in dtype(self)
   1609             formats.append(dt)
   1610 
-> 1611         return np.dtype({'names': self.names,
   1612                          'formats': formats,
   1613                          'offsets': offsets})

ValueError: field names must be strings

topcat appears to be able to read and write the file without issue.

Expected behavior

astropy.table saves and loads the FITS tables correctly.

Actual behavior

astropy.table fails to save correctly FITS tables containing a column named “a: 0” and cannot load the generated file.

Steps to Reproduce

The following code reproduces the issue for “a: 0”.

import uuid

from astropy.table import Table, Column

def test_bug(string):
    fname = f"{uuid.uuid4()}.fits"
    table = Table()
    table.add_column(Column([0.], name=string))
    table.write(fname)
    try:
        Table.read(fname)
        print(f"{fname}: success ({string})")
    except:
        print(f"{fname}: failure ({string})")

test_bug("a:b")
test_bug("a: b")
test_bug("a :b")
test_bug("a : b")

test_bug("a:0")
test_bug("a: 0")
test_bug("a :0")
test_bug("a : 0")

test_bug("0:a")
test_bug("0: a")
test_bug("0 :a")
test_bug("0 : a")

test_bug("0:0")
test_bug("0: 0")
test_bug("0 :0")
test_bug("0 : 0")

2ba9e3e9-f280-452d-80f2-fd9f7bcc985c.fits: success (a: b)
1b60e2a6-234c-46da-9ea5-8ba66091713e.fits: success (a :b)
0dffea9b-85a0-4185-8d90-736d2786095e.fits: success (a : b)
9c4bef60-cdcc-4e46-b98c-1642081d7914.fits: success (a:0)
8d4584eb-4f4b-4215-b138-5070614b49f7.fits: failure (a: 0)
59c79405-e8d7-4615-838c-00c8ffab94ca.fits: success (a :0)
6650b1d3-f452-481f-8234-8a526c6cb424.fits: success (a : 0)
30bbd742-96b0-4565-b228-f7159e69befe.fits: success (0:a)
fbc7f0ba-859f-4f6d-9e2d-b8c21bb11f97.fits: success (0: a)
ad2f16ca-488a-4620-aba4-efd1244eab19.fits: success (0 :a)
8df53f1d-60f3-45fc-ba1e-3024e20c9e96.fits: success (0 : a)
ea455afd-8cff-497f-89f7-6b032431f3ee.fits: success (0:0)
8e3f2d44-68bd-40bd-b02f-0ab34a6a73a5.fits: success (0: 0)
ffa7fd2c-0a99-4c30-8700-7b0bd25bc7c1.fits: success (0 :0)

System Details

Linux-5.6.6-arch1-1-x86_64-with-glibc2.2.5 Python 3.8.2 (default, Apr 8 2020, 14:31:25) [GCC 9.3.0] Numpy 1.18.3 astropy 4.0.1 Scipy 1.4.1

About this issue

  • Original URL
  • State: open
  • Created 4 years ago
  • Reactions: 1
  • Comments: 17 (17 by maintainers)

Most upvoted comments

Issuing the warning by changing this to TTYPE_RE = re.compile(r'[0-9a-zA-Z_]+$') hopefully will not break anything, but the core of the actual problem seems to be in https://github.com/astropy/astropy/blob/49a762a4be2b9f580d17b57ae524c376f8afb33e/astropy/io/fits/card.py#L602-L623 which explicitly supports assigning cards in some str("KEY = FIELD-SPECIFIER: VALUE") syntax – not sure exactly what card the example is supposed to generate, but a name like "a: 0" clearly is bound to mess with that. And it might be very tricky to fix that without breaking existing code.