lightkurve: Memory error - KeplerTargetPixelFile iteration

Problem description

When iterate over KeplerTargetPixelFile object clogging of memory occure.

Example

from lightkurve import KeplerTargetPixelFile

tpf = KeplerTargetPixelFile(filename)
for i, frame in enumerate(tpf):
    pass

Behavior

Over 1,000 megabytes ram for i = 250. This same behavior for jupyter, ipython, python sheel.

Environment:

platform Linux
lightkurve version: 1.0b21
installation method: conda

About this issue

Original URL
State: open
Created 6 years ago
Reactions: 1
Comments: 20 (5 by maintainers)

Most upvoted comments

Haven’t been around lightkurve much lately because of commitments to cryptography projects. But, memory leaks are critical issues. When I have time, I’ll look at the tests some more.

texadactyl on Sep 5, 2020

FWIW adapting the mprof script to this example (BTW with Python 3.11.4 under macOS 12.7 on M1 (arm64) CPU, so quite different setting)

#!/usr/bin/env python
import numpy as np
import astropy
import lightkurve as lk
from memory_profiler import profile

@profile
def gettpf(ticid):
    return lk.search_targetpixelfile(f'TIC {ticid}').download_all()

if __name__ == '__main__':
    print(f'numpy {np.__version__}')
    print(f'astropy {astropy.version.version}')
    print(f'lightkurve {lk.__version__}')
    print(80 * '-')
    ticid = 85148268

    for i in range(10):
        tpf = gettpf(ticid)
        print(i, ticid, tpf[0].shape, np.size(tpf[0].flux))
        del tpf

I find this memory usage for the first couple iterations with 2.4.0:

numpy 1.25.2
astropy 5.3.3
lightkurve 2.4.0
--------------------------------------------------------------------------------
Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7    240.1 MiB    240.1 MiB           1   @profile
     8                                         def gettpf(ticid):
     9    526.2 MiB    286.1 MiB           1       return lk.search_targetpixelfile(f'TIC {ticid}').download_all()

0 85148268 (18882, 55, 11) 11423610
Filename: tpf_leak-2.4.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7    335.6 MiB    335.6 MiB           1   @profile
     8                                         def gettpf(ticid):
     9    613.5 MiB    277.9 MiB           1       return lk.search_targetpixelfile(f'TIC {ticid}').download_all()

1 85148268 (18882, 55, 11) 11423610
Filename: tpf_leak-2.4.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7    335.7 MiB    335.7 MiB           1   @profile
     8                                         def gettpf(ticid):
     9    613.5 MiB    277.9 MiB           1       return lk.search_targetpixelfile(f'TIC {ticid}').download_all()

in comparison with 2.4.1

numpy 1.25.2
astropy 5.3.3
lightkurve 2.4.1
--------------------------------------------------------------------------------
Filename: tpf_leak-2.4.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7    241.1 MiB    241.1 MiB           1   @profile
     8                                         def gettpf(ticid):
     9    805.5 MiB    564.4 MiB           1       return lk.search_targetpixelfile(f'TIC {ticid}').download_all()


0 85148268 (18882, 55, 11) 11423610
Filename: tpf_leak-2.4.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7    805.5 MiB    805.5 MiB           1   @profile
     8                                         def gettpf(ticid):
     9   1083.8 MiB    278.3 MiB           1       return lk.search_targetpixelfile(f'TIC {ticid}').download_all()


1 85148268 (18882, 55, 11) 11423610
Filename: tpf_leak-2.4.py

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
     7   1083.8 MiB   1083.8 MiB           1   @profile
     8                                         def gettpf(ticid):
     9   1361.8 MiB    278.0 MiB           1       return lk.search_targetpixelfile(f'TIC {ticid}').download_all()

and so on, continuing to accumulate some ¼ GB memory usage per iteration, i.e. apparently the memory from each newly loaded tpf is never released again (does not matter whether I run del tpf or not, or if I only download a single tpf with .download() – in the latter case the increment is just reduced to ~226 MB).

Probably not related to the original bug reported here, but definitely an issue!

dhomeier on Sep 29, 2023