MINGW-packages: [python-tifffile] unable to write bigtiff file (hangs at 4GB)

Description / Steps to reproduce the issue

Unable to write TIFF file larger than 4GB.

After posting on Discord (https://discord.com/channels/792780131906617355/1078393517396729886), I was asked to open an issue on github.

I am trying to write a large TIFF file (bigtiff) using imwrite from tifffile in python. The file writes up to about 4GB and hangs. I feel like this might be based on how libtiff was built, maybe? The type of TIFF I am writing is a numpy array that is about 100000 x 100000 and is and 8 bit grayscale (unit8).

I have tried to do the imwrite on a linux machine and the same code works fine. And indeed, a colleague with a Windows conda environment also reports success.

I came across this : https://gist.github.com/jcupitt/ba60e4e1100607d7a5cc9c19e2ec11e8 which seems to indicate some special build flag/config for large files, and wondered if it might be relevant?

Here is example of the imwrite:

imwrite("test.tiff", neg, 
        photometric='minisblack', dtype='uint8')

TIFF info:

$ pacman -Q | grep -i tiff
mingw-w64-x86_64-libtiff 4.5.0-1
mingw-w64-x86_64-python-tifffile 2023.2.3-1

Expected behavior

I expect that the file would write successfully.

Actual behavior

The write starts successfully, but “hangs” at 4GB.

Verification

I have verified that my MSYS2 is up-to-date before submitting the report (see https://www.msys2.org/docs/updating/)

Windows Version

MINGW64_NT-10.0-19042

MINGW environments affected

MINGW64
MINGW32
UCRT64
CLANG64
CLANG32
CLANGARM64

Are you willing to submit a PR?

No. I wouldn’t now where to start. 😉

About this issue

Original URL
State: closed
Created a year ago
Comments: 22 (7 by maintainers)

Most upvoted comments

Keep in mind Cygwin’s gcc defines _WIN64

it does not

lazka on Mar 31, 2023

Thanks to everyone for getting this fixed. This was my first experience doing a bug report or anything github-ish. All in all, a pleasant experience.

rlbrott on Apr 6, 2023

Keep in mind Cygwin’s gcc defines _WIN64

it does not

Well, that was dumb. For some formerly valid reason I still had /mingw64/bin in my main system environment PATH, I have now rectified that with extreme prejudice. Thanks for saying something @lazka.

Although, as surprising as it was to have _WIN64 (especially with no gcc installed in the first place), I find it just as surprising that Cygwin doesn’t “sanitize” its startup environment to get a more consistent OOB *nix experience. Isn’t that the point of Cygwin, even more than MSYS2? Whatever, I don’t really know what I’m talking about.

cbrt64 on Apr 3, 2023

Further testing indicates that the problem is in numpy itself because numpy.save() also hangs. I looked at the numpy source and there is a function PyArray_ToFile in numpy/core/src/multiarray/convert.c which seems to be the workhorse. There is a reference to numpy issue 1660 in the code for a workaround to write large files for Win64 at about line 160 or so. When I build numpy, this check didn’t seem to be triggered, so I forced it to compile that section and tofile() worked. I am not sure the right way to fix this, however.

rlbrott on Feb 28, 2023