astropy: FITS tables with a column “a: 0” produce duplicate TTYPE entries
Description
astropy.table fails to save correctly FITS tables containing a column named “a: 0” and cannot load the generated file. This occurs when the column name starts with a letter, then contains a colon, then a space, then a number. In this case the TTYPE entry appears to be duplicated. For instance with the above example the FITS header generated by astropy.table is the following:
XTENSION= ‘BINTABLE’ / binary table extension
BITPIX = 8 / array data type
NAXIS = 2 / number of array dimensions
NAXIS1 = 8 / length of dimension 1
NAXIS2 = 1 / length of dimension 2
PCOUNT = 0 / number of group parameters
GCOUNT = 1 / number of groups
TFIELDS = 1 / number of table fields
TTYPE1 = ‘a: 0’
TTYPE1 = ‘a: 0’
TFORM1 = 'D ’
TTYPE1 = ‘a: 0’
Attempting to read this file with astropy.table leads to a ValueError exception:
ValueError Traceback (most recent call last)
<ipython-input-2-b76ff7c17fe6> in <module>
----> 1 t = Table.read('8d4584eb-4f4b-4215-b138-5070614b49f7.fits')
/usr/lib/python3.8/site-packages/astropy/table/connect.py in __call__(self, *args, **kwargs)
50 def __call__(self, *args, **kwargs):
51 cls = self._cls
---> 52 out = registry.read(cls, *args, **kwargs)
53
54 # For some readers (e.g., ascii.ecsv), the returned `out` class is not
/usr/lib/python3.8/site-packages/astropy/io/registry.py in read(cls, format, *args, **kwargs)
521
522 reader = get_reader(format, cls)
--> 523 data = reader(*args, **kwargs)
524
525 if not isinstance(data, cls):
/usr/lib/python3.8/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
213
214 try:
--> 215 return read_table_fits(hdulist, hdu=hdu,
216 astropy_native=astropy_native)
217 finally:
/usr/lib/python3.8/site-packages/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
226 # In the loop below we access the data using data[col.name] rather than
227 # col.array to make sure that the data is scaled correctly if needed.
--> 228 data = table.data
229
230 columns = []
/usr/lib/python3.8/site-packages/astropy/utils/decorators.py in __get__(self, obj, owner)
756 return val
757 else:
--> 758 val = self.fget(obj)
759 obj.__dict__[self._key] = val
760 return val
/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py in data(self)
397 @lazyproperty
398 def data(self):
--> 399 data = self._get_tbdata()
400 data._coldefs = self.columns
401 data._character_as_bytes = self._character_as_bytes
/usr/lib/python3.8/site-packages/astropy/io/fits/hdu/table.py in _get_tbdata(self)
169 type=np.rec.recarray)
170 else:
--> 171 raw_data = self._get_raw_data(self._nrows, columns.dtype,
172 self._data_offset)
173 if raw_data is None:
/usr/lib/python3.8/site-packages/astropy/utils/decorators.py in __get__(self, obj, owner)
756 return val
757 else:
--> 758 val = self.fget(obj)
759 obj.__dict__[self._key] = val
760 return val
/usr/lib/python3.8/site-packages/astropy/io/fits/column.py in dtype(self)
1609 formats.append(dt)
1610
-> 1611 return np.dtype({'names': self.names,
1612 'formats': formats,
1613 'offsets': offsets})
ValueError: field names must be strings
topcat appears to be able to read and write the file without issue.
Expected behavior
astropy.table saves and loads the FITS tables correctly.
Actual behavior
astropy.table fails to save correctly FITS tables containing a column named “a: 0” and cannot load the generated file.
Steps to Reproduce
The following code reproduces the issue for “a: 0”.
import uuid
from astropy.table import Table, Column
def test_bug(string):
fname = f"{uuid.uuid4()}.fits"
table = Table()
table.add_column(Column([0.], name=string))
table.write(fname)
try:
Table.read(fname)
print(f"{fname}: success ({string})")
except:
print(f"{fname}: failure ({string})")
test_bug("a:b")
test_bug("a: b")
test_bug("a :b")
test_bug("a : b")
test_bug("a:0")
test_bug("a: 0")
test_bug("a :0")
test_bug("a : 0")
test_bug("0:a")
test_bug("0: a")
test_bug("0 :a")
test_bug("0 : a")
test_bug("0:0")
test_bug("0: 0")
test_bug("0 :0")
test_bug("0 : 0")
2ba9e3e9-f280-452d-80f2-fd9f7bcc985c.fits: success (a: b)
1b60e2a6-234c-46da-9ea5-8ba66091713e.fits: success (a :b)
0dffea9b-85a0-4185-8d90-736d2786095e.fits: success (a : b)
9c4bef60-cdcc-4e46-b98c-1642081d7914.fits: success (a:0)
8d4584eb-4f4b-4215-b138-5070614b49f7.fits: failure (a: 0)
59c79405-e8d7-4615-838c-00c8ffab94ca.fits: success (a :0)
6650b1d3-f452-481f-8234-8a526c6cb424.fits: success (a : 0)
30bbd742-96b0-4565-b228-f7159e69befe.fits: success (0:a)
fbc7f0ba-859f-4f6d-9e2d-b8c21bb11f97.fits: success (0: a)
ad2f16ca-488a-4620-aba4-efd1244eab19.fits: success (0 :a)
8df53f1d-60f3-45fc-ba1e-3024e20c9e96.fits: success (0 : a)
ea455afd-8cff-497f-89f7-6b032431f3ee.fits: success (0:0)
8e3f2d44-68bd-40bd-b02f-0ab34a6a73a5.fits: success (0: 0)
ffa7fd2c-0a99-4c30-8700-7b0bd25bc7c1.fits: success (0 :0)
System Details
Linux-5.6.6-arch1-1-x86_64-with-glibc2.2.5 Python 3.8.2 (default, Apr 8 2020, 14:31:25) [GCC 9.3.0] Numpy 1.18.3 astropy 4.0.1 Scipy 1.4.1
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 1
- Comments: 17 (17 by maintainers)
Issuing the warning by changing this to
TTYPE_RE = re.compile(r'[0-9a-zA-Z_]+$')
hopefully will not break anything, but the core of the actual problem seems to be in https://github.com/astropy/astropy/blob/49a762a4be2b9f580d17b57ae524c376f8afb33e/astropy/io/fits/card.py#L602-L623 which explicitly supports assigning cards in somestr("KEY = FIELD-SPECIFIER: VALUE")
syntax – not sure exactly what card the example is supposed to generate, but a name like"a: 0"
clearly is bound to mess with that. And it might be very tricky to fix that without breaking existing code.