astropy: Segfault with CompImageHDU

The following code (sometimes) segfaults on my machine.

import numpy as np
from astropy.io import fits
from io import BytesIO
import faulthandler

faulthandler.enable()

arr = np.array([1,5123124,2,1,2,1]).astype(int)
hdu = fits.CompImageHDU(arr, compression_type="PLIO_1")
buffer = BytesIO()
hdu.writeto(buffer)
print(buffer.getvalue())
buffer.seek(0)
with fits.open(buffer, memmap=False) as hdulist:
    print(hdulist)

The faulthandler reports varying stack-traces, for example:

Current thread 0x0000344c (most recent call first):
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1663 in _update_compressed_data
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1793 in _prewriteto
  File "...\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 935 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 1582 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper

Current thread 0x000000f4 (most recent call first):
  File "...\lib\site-packages\astropy\io\fits\util.py", line 674 in _array_to_file_like
  File "...\lib\site-packages\astropy\io\fits\util.py", line 651 in _array_to_file
  File "...\lib\site-packages\astropy\io\fits\file.py", line 380 in writearray
  File "...\lib\site-packages\astropy\io\fits\hdu\table.py", line 931 in _writedata_internal
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 610 in _writedata
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1839 in _writedata
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 680 in _writeto_internal
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 674 in _writeto
  File "...\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 936 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 1582 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper

Current thread 0x00002a60 (most recent call first):
  File "...\lib\site-packages\astropy\io\fits\column.py", line 1953 in __new__
  File "...\lib\site-packages\astropy\io\fits\fitsrec.py", line 782 in _convert_p
  File "...\lib\site-packages\astropy\io\fits\fitsrec.py", line 702 in field
  File "...\lib\site-packages\astropy\io\fits\hdu\table.py", line 522 in _prewriteto
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1821 in _prewriteto
  File "...\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 935 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 1582 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper

It could be windows-specific as in PR #9641 but could also be something different.

It could be an issue with an upstream dependency like NumPy, cfitsio or even CPython. I wasn’t able to get to the root-cause of the issue.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 16 (16 by maintainers)

Most upvoted comments

So is this bug still in cfitsio or in astropy.io.fits?

for what it’s worth at this point (when we are going to fix it in a soon-to-be opened pr), but this is a bug in astropy.io.fits.

Cadair on Dec 15, 2022

Or we’re passing invalid parameters… I really hate C debugging in docker 😟

MSeifert04 on Dec 4, 2019

Or inside CFITSIO 🤷‍♂

MSeifert04 on Dec 3, 2019

In my linux docker environment with python&numpy debug:

free(): invalid next size (fast)
Fatal Python error: Aborted

Current thread 0x00007fa2ce726740 (most recent call first):
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/compressed.py", line 1663 in _update_compressed_data
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/compressed.py", line 1793 in _prewriteto
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py", line 935 in writeto
  File "/usr/local/lib/python3.8/site-packages/astropy/utils/decorators.py", line 521 in wrapper
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py", line 1581 in writeto
  File "/usr/local/lib/python3.8/site-packages/astropy/utils/decorators.py", line 521 in wrapper
  File "/io/test1.py", line 11 in <module>
Aborted

So probably not windows-specific.

MSeifert04 on Dec 3, 2019

Well, it could be windows-specific but I haven’t had time to run it in docker 😄

MSeifert04 on Dec 3, 2019

On further inspection this probably isn’t related to #9641 because it doesn’t fail for non-PLIO compressions. 🤷‍♂

MSeifert04 on Dec 3, 2019