astropy: Unit parsing errors produce misleading advice in certain cases

Description

When certain units cannot be parsed, a specific message is printed that can be misleading in certain cases. If a user follows the advice, the result is an exception.

Expected behavior

The warning about defining a new unit in order to allow it to be parsed should take into account units that are already defined. In addition, astropy.units should go ahead and parse the unit anyway, if it is already defined in astropy.units.

How to Reproduce

  1. Set up a FITS file.
>>> import numpy as np
>>> import astropy.units as u
>>> from astropy.io import fits
>>> from astropy.table import Table, QTable
>>> rng = np.random.default_rng()
>>> targetid = rng.integers(0, 2**64, (20, ), dtype=np.uint64)
>>> mag = 1e9 * rng.random((20, ), dtype=np.float32)
>>> columns = fits.ColDefs([fits.Column(name='TARGETID', format='K', bzero=2**63, array=targetid),
>>>                         fits.Column(name='FLUX', format='E', array=mag)])
>>> hdu0 = fits.PrimaryHDU()
>>> hdu0.header.append(('EXTNAME', 'PRIMARY', 'extension name'))
>>> hdu0.add_checksum()
>>> hdu1 = fits.BinTableHDU.from_columns(columns, name='BINTABLE', uint=True)
>>> hdu1.header.insert('TFORM2', ('TUNIT2', 'nanomaggy', 'Units for FLUX'), after=True)
>>> hdu1.add_checksum()
>>> hdulist = fits.HDUList([hdu0, hdu1])
>>> hdulist.writeto('demonstrate_maggy.fits', overwrite=True)
  1. Try to read the FITS file.
>>> t = Table.read('demonstrate_maggy.fits', hdu='BINTABLE')
WARNING: UnitsWarning: 'nanomaggy' did not parse as fits unit: At col 0, Unit 'nanomaggy' not supported by the FITS standard.  If this is meant to be a custom unit, define it with 'u.def_unit'. To have it recognized inside a file reader or other code, enable it with 'u.add_enabled_units'. For details, see https://docs.astropy.org/en/latest/units/combining_and_defining.html [astropy.units.core]
  1. Great, I’ll define the unit then!
>>> nanomaggy = u.def_unit('nanomaggy', 1.0e-9 * 3631*u.Jy)
>>> u.add_enabled_units( [nanomaggy,])
Traceback (most recent call last):
  File "/Users/weaver/Documents/Code/git/desihub/desiutil/demonstrate_maggy.py", line 27, in <module>
    u.add_enabled_units( [nanomaggy,])
  File "/Users/weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/astropy/units/core.py", line 469, in add_enabled_units
    get_current_unit_registry().add_enabled_units(units)
  File "/Users/weaver/Documents/local/products/venv/desiconda/lib/python3.10/site-packages/astropy/units/core.py", line 221, in add_enabled_units
    raise ValueError(
ValueError: Object with name 'nanomaggy' already exists in namespace. Filter the set of units to avoid name clashes before enabling them.
  1. OK, so if nanomaggy is already defined, why didn’t you say so? I can understand the FITS warning, but astropy.units clearly has the capability to parse it anyway.

Versions

import platform; print(platform.platform())
import sys; print("Python", sys.version)
import astropy; print("astropy", astropy.__version__)
import numpy; print("Numpy", numpy.__version__)
import erfa; print("pyerfa", erfa.__version__)
import scipy; print("Scipy", scipy.__version__)
import matplotlib; print("Matplotlib", matplotlib.__version__)
macOS-13.5.2-x86_64-i386-64bit
Python 3.10.10 (main, Feb 10 2023, 08:40:50) [Clang 14.0.0 (clang-1400.0.29.202)]
astropy 5.2.1
Numpy 1.22.4
pyerfa 2.0.0.1
Scipy 1.8.1
Matplotlib 3.6.2

About this issue

  • Original URL
  • State: open
  • Created 10 months ago
  • Comments: 16 (16 by maintainers)

Most upvoted comments

Taking a look now. Let me delve into the secondary bug at first (I’m running a very slightly simplified and self-contained version of the MWE, reproduced hereafter for convenience)

import numpy as np
import astropy.units as u
from astropy.table import Table
from astropy.io import fits

rng = np.random.default_rng()
targetid = rng.integers(0, 2**64, (1, ), dtype=np.uint64)
mag = 1e9 * rng.random((1,), dtype=np.float32)
columns = fits.ColDefs([fits.Column(name='TARGETID', format='K', bzero=2**63, array=targetid),
                        fits.Column(name='FLUX', format='E', array=mag)])
hdu0 = fits.PrimaryHDU()
hdu0.header.append(('EXTNAME', 'PRIMARY', 'extension name'))
hdu0.add_checksum()
hdu1 = fits.BinTableHDU.from_columns(columns, name='BINTABLE', uint=True)
hdu1.header.insert('TFORM2', ('TUNIT2', 'nanomaggy', 'Units for FLUX'), after=True)
hdu1.add_checksum()
hdulist = fits.HDUList([hdu0, hdu1])
hdulist.writeto('demonstrate_maggy.fits', overwrite=True)

t1 = Table.read(hdu1, hdu='BINTABLE')
t2 = Table.read('demonstrate_maggy.fits', hdu='BINTABLE')

printing t1 and t2 reveals that t1 seems to have lost units metadata:

>>> print(t1)
      TARGETID           FLUX   
       uint64          float32  
-------------------- -----------
11049962027143826759 743222300.0

>>> print(t2)
      TARGETID           FLUX   
                      nanomaggy 
       uint64          float32  
-------------------- -----------
11049962027143826759 743222300.0

so I guess that’s another symptom of the suspected bug, and maybe more easily exploitable than the difference between getting a warning or not. At a high level, the difference is that, when passed a filename, read_table_fits will first read it as a HDUList, then recurse https://github.com/astropy/astropy/blob/48a792f9d8dc659f428e9c6f0e9e9964a1b60682/astropy/io/fits/connect.py#L234-L252 In the nested call, the following special-case code will be run https://github.com/astropy/astropy/blob/48a792f9d8dc659f428e9c6f0e9e9964a1b60682/astropy/io/fits/connect.py#L184 None of this happens if you just pass in a BinTableHDU, as is evident from the first listing. Deeper investigation is needed, but I think this bug (and discussion) deserves its own issue, so I’ll open a separate one. I also intend to come back and study the main bug reported here more closely.

update: xref #16077

I guess the “connect” logic in io.fits to Table could catch this warning and and throw another warning that is more useful? 🤔

An additional test:

  1. I created a FITS file exactly like the example above except I replaced nanomaggy with nanofluxy.
  2. Before attempting to read the FITS file at all, I added nanofluxy to astropy.units:
>>> nanofluxy = u.def_unit('nanofluxy', 1.0e-9*u.Jy)
>>> u.add_enabled_units( [nanofluxy,])
>>> t = Table.read('demonstrate_maggy.fits', hdu='BINTABLE')
  1. I get the same warning message. That’s not surprising, because the warning message is hard-coded.
  2. For someone who doesn’t know that the warning message is hard-coded, then the message would be a surprise, e.g. “I just defined that unit, why am I still getting this message about the unit being undefined?”

Interesting. I also want to test what happens with some other unit that isn’t already defined, I’ll give that a try tomorrow.

Correct, nanomaggy is not part of the FITS standard. Nevertheless, it is defined by astropy.units.photometric. I do think it is fair to issue a warning about a unit not being in the FITS standard.

However, it’s the second part of the warning that is misleading here, and that’s why I’m reporting this.