astropy: Error reading a FITS table into a pandas DataFrame via astropy

I’m trying to read a FITS table into a pandas DataFrame via astropy.

Test data and script is here: https://github.com/cdeil/test_fits_to_pandas

This is the error I get (same error from read_via_table.py):

$ python read_via_fits.py 
                                       nfev statname  numpoints
ROI                       Solution                             
HESS_J1023m575            4_big         273     cash      75000
                          5             316     cash      75000
                          3             131     cash      75000
                          4             273     cash      75000
                          2_elongated   111     cash      75000
ERROR: ValueError: Big-endian buffer not supported on little-endian compiler [unknown]
Traceback (most recent call last):
  File "read_via_fits.py", line 8, in <module>
    print(table[['nfev', 'statname']])
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2017, in __getitem__
    return self._getitem_array(key)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2054, in _getitem_array
    return self.take(indexer, axis=1, convert=True)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2998, in take
    return self.reindex(columns=new_columns)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2630, in reindex
    fill_value, limit)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2722, in _reindex_columns
    copy, fill_value)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2739, in _reindex_with_indexers
    fill_value=fill_value)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/internals.py", line 1959, in reindex_indexer
    return self._reindex_indexer_items(new_axis, indexer, fill_value)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/internals.py", line 1993, in _reindex_indexer_items
    allow_fill=False)
  File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/common.py", line 540, in take_nd
    func(arr, indexer, out, fill_value)
  File "generated.pyx", line 2842, in pandas.algos.take_2d_axis0_int16_int16 (pandas/algos.c:69040)
ValueError: Big-endian buffer not supported on little-endian compiler

@iguananaut @astrofrog Am I doing something wrong (passing a FITSrec instead of a recarray)? Is there a workaround?

About this issue

Original URL
State: closed
Created 11 years ago
Comments: 29 (26 by maintainers)

Most upvoted comments

So as per the advice in #3778 the solution is to call array.byteswap().newbyteorder() before passing the structured array to the pandas DataFrame constructor.

Is it worth adding astropy.table.Table.to_dataframe() and astropy.table.Table.from_dataframe() methods to make it easy to go back and forth between astropy Table and pandas DataFrame and take care of the byte swapping (and mask, I guess there’s no way to preserve the FITS header info) there? Would this be a welcome addition and acceptable to have a pandas import in that method? And if you don’t think it’s worth it: how about adding this info to the astropy docs somewhere?

cdeil on Jun 6, 2013

There’s a fair bit of code out there in SciPy, and I’m guessing Pandas as well, that assumes all the data it’s going to get is little-endian, or at least doesn’t bother to byteswap automatically where necessary. So that wouldn’t surprise me.

You’d actually be amazed by the amount of energy that has been put into debating whether or not FITS’ big-endian requirement is harmful. I and others argue that it is, some argue that it’s a silly thing to worry about. I think both sides have good points, but I’m quite devoted to my opinion on the matter and so are they. (I feel though that the big-endian adherents just haven’t had as much experience dealing with the real-world annoyances that arise as a result…)

embray on Feb 18, 2015