astropy: HIERARCH keywords don't work with CONTINUE cards

The astropy.io.fits manual states, that we can use header keywords longer than 8-characters. In this case HIERARCH cards will be created. The manual also states, that if we want to store keyword-value pairs longer than 80-characters, continue cards will automatically be created.

However, in practice it seems that both definitions work only mutually exclusive, i.e. we can not create a FITS file containing a keyword value pair, where the keyword is longer than 8-characters (i.e. a HIERARCH keyword) and the value is a very long string.

An example:

from astropy.io import fits

header1 = fits.Header()
header2 = fits.Header()
header3 = fits.Header()

header1['TEST'] = 'superlongstring'*10
header2['TEST TEST'] = 'superlongstring'
header3['TEST TEST'] = 'superlongstring'*10

Here header1 and header2 will be correct, but when calling repr(header3) or attempting to save a FITS file with such a header, the error ValueError: The keyword TEST TEST with its value is too long is raised.

Looking at the astropy source code, it is explicitly ruled out to use CONTINUE cards in a value belonging to a HIERARCH keyword.

In the source code we find (astropy/io/fits/card.py#L1227-1236):

keywordvalue_length = len(keyword) + len(delimiter) + len(value)
if (keywordvalue_length > self.length and
        keyword.startswith('HIERARCH')):
    if (keywordvalue_length == self.length + 1 and keyword[-1] == ' '):
        output = ''.join([keyword[:-1], delimiter, value, comment])
    else:
        # I guess the HIERARCH card spec is incompatible with CONTINUE
        # cards
        raise ValueError('The keyword %s with its value is too long' %
                         self.keyword)

The comment is from 2011, and was put there during a redesign of pyfits. Before that, pyfits could also read or write exlusively either HIERARCH cards or cards with CONTINUE statement, and apparently it was kept like that. The relevant old PyFITS code is:

if cardimage[:8].upper() == 'HIERARCH':
    card = _HierarchCard()
    # for card image longer than 80, assume it contains CONTINUE card(s).
elif len(cardimage) > Card.length:
    card = _ContinueCard()

Both, the HIERARCH and the CONTINUE keywords are not part of the official FITS standard (Pence, W.D. et al. 2010, Astronomy & Astrophysics Vol. 524, A42). The HIERARCH keyword convention is given in Wiecencec, A. et al. 2009, “The ESO HIERARCH Keyword Convention”) and the CONTINUE convention is given by the HEASARC FITS Working Group, 2007. “The CONTINUE Long String Keyword Convention”. Reading carefully both of these definitions, I see absolutely no reason why they should be mutually exclusive. Hence, astropy should support HIERARCH keywords with CONTINUE values.

About this issue

Original URL
State: open
Created 9 years ago
Comments: 20 (17 by maintainers)

Commits related to this issue

* Adding a `write_fits` method for the `Camera` class that wraps the `panoptes-utils` version and sets the `_readout_complete` (i.e. `waiting_for_ready` property). * Fixing issue relate to FITS header... — committed to wtgee/POCS by wtgee 3 years ago
MLO Updates (#1150) * DSLR camera simulator * Don't clear exposing event in simulator as it's handled in the base class. * * Change NTP router ip * * Better support and consistent use of `en... — committed to panoptes/POCS by wtgee 2 years ago

Most upvoted comments

I just ran into this issue. Here is a somewhat practical example:

import os
from astropy.io import fits

if os.path.isfile('test.fits'):
    os.remove('test.fits')

# This is saving the full path to a file to the header and works fine
# because it doesn't need to use both HIERARCH and CONTINUE
hdr = fits.Header()
hdr['MASKFILE'] = '/home/xavier/Projects/PypeIt-development-suite/RAW_DATA/keck_deimos/830G_M_8500/DE.20100913.57006.fits.gz'
# Printing is fine
print(hdr)
# Keyword selection is fine
print(hdr['MASKFILE'])
# Writing to a file is fine
fits.HDUList([fits.PrimaryHDU(header=hdr)]).writeto('test.fits', overwrite=True)
hdu = fits.open('test.fits')
# And it's appropriately recovered
print(hdu[0].header['MASKFILE'])

# Here, I'm using what we originally had for the header keyword.
# There's a rather trivial work around as shown above; this is just to
# illustrate the problem.

hdr = fits.Header()
# Defining the header card doesn't cause a fault
hdr['MASKDEF_FILE'] = '/home/xavier/Projects/PypeIt-development-suite/RAW_DATA/keck_deimos/830G_M_8500/DE.20100913.57006.fits.gz'
# And you can print the value
print(hdr['MASKDEF_FILE'])

try:
    # But printing the whole header faults
    print(hdr)
except Exception as e:
    print(str(e))

try:
    # As does trying to write it to a file
    fits.HDUList([fits.PrimaryHDU(header=hdr)]).writeto('test.fits', overwrite=True)
except Exception as e:
    print(str(e))

if os.path.isfile('test.fits'):
    os.remove('test.fits')

kbwestfall on Dec 1, 2020

I agree that strictly spoken the CONTINUE extension is to be applied (only) to string values, while HIERARCH headers have no value (in the standard FITS meaning) at all.

However, when reading both conventions (HIERARCH and CONTINUE), I would guess it is more an extension to CONTINUE than to HIERARCH. However, in my opinion the CONTINUE extension is made in a way that extending it would be compatible with all possible cases of FITS reader. Specifically, the CONTINUE extension says:

If a FITS reader encounters a CONTINUE keyword that is not preceded by a string keyword (or another CONTINUE keyword) whose value string ends with the `&’ character, then that CONTINUE keyword should be ignored (i.e., it should be interpreted the same as a COMMENT keyword).

Lets look for a possible header:

HIERARCH ASTROPY EXAMPLE = 'this is a very &'
CONTINUE 'long string value'

For the possible cases of standard conform FITS readers this means:

FITS reader that don’t support HIERARCH and also not CONTINUE: They will ignore all HIERARCH and CONTINUE values, as expected;
FITS reader that support HIERARCH, but not CONTINUE: they will shorten all strings (be them HIERARCH or not) to the first line, which is what one would expect,
FITS reader that support CONTINUE, but not HIERARCH: They will ignore the CONTINUE line after a HIERARCH line (and the HIERARCH line itself), which is again what expected
FITS reader that support mutually both keywords mutually exclusive: They will read the first line for HIERARCH keywords, but ignore a following CONTINUE card, again what one would expect
A hypothetical FITS reader that supports both keywords at the same time, would get the full string.

Also, the extension to support CONTINUE cards after HIERARCH cards would be quite natural, there is not much danger that someone comes up with a alternative idea. Therefore, I could imagine that it would be not a mistake to just support this (maybe when a special flag is set). There is the danger that the long value may not be read by other readers, but there is no danger of misinterpretation.

olebole on May 1, 2015