astropy: download_files_in_parallel with cache=False doesn't work on Windows

In #6658 I had problems locally with cache=False. For example:

from astropy.utils.data import download_files_in_parallel
import os
dwn = download_files_in_parallel(['http://www.google.de', 'http://www.wikipedia.org'])
assert all([os.path.isfile(f) for f in dwn]), dwn

This fails locally - while it works if one just adds cache=True:

from astropy.utils.data import download_files_in_parallel
import os
dwn = download_files_in_parallel(['http://www.google.de', 'http://www.wikipedia.org'], cache=True)
assert all([os.path.isfile(f) for f in dwn]), dwn

Could be a Windows-thing with subprocesses that remove temp-files created upon exit. But I don’t have the experience to make more than guesses. Would be interesting to know if that’s a fundamental problem with that function or just a Windows thing.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 22 (22 by maintainers)

Most upvoted comments

could you check if the exit handler is called when the sub-processes terminate ?

Yes. On Windows, with cache=False, when I insert a print statement right before os.remove there, I see both downloaded files deleted (even before the progress bar reports that download is finished).

When cache=True, it is still called but nothing is done because _tempfilestodel is an empty list.

Maybe this is beyond our control and we should just document this behavior in the API?

pllim on Oct 5, 2017

Well it uses multiprocessing to download the files. Probably uses separate processes that clean up the files on Windows and ones that share the temp-files with the main process on *NIX systems.

MSeifert04 on Oct 5, 2017