esp-idf: spiffsgen.py under windows does not generate valid binary images (IDFGH-4615)

Environment

• Module or chip used: ESP32-WROVER-IB, 16MB flash, 4MB SPIRAM
• IDF version (run git describe --tags to find it): release/v4.1
• Build System: CMake + idf.py
• Compiler version gcc for release/v4.1
• Operating System: Windows
• (Windows only) environment type: MSYS2 mingw32 & mingw64
• Using an IDE?: No
• Power Supply: external 3.3V

Problem Description

spiffsgen.py on windows builds corrupt image from directory, when this is flashed firmware doesn’t run as some files in spiffs partition are missing or corrupt

diff storage_ok.bin storage_bad.bin (attached good (Linux + MacOS) and bad images (Windows)) indicates erroneous image as files differ, storage_ok.bin was correctly built on Mac / Linux

Expected Behavior

see attached storage_ok.bin in storage.zip see attached spiffs_image.zip file which contains desired file structure (compressed for this issue request)

Actual Behavior

see attached storage_bad.bin in storage.zip bad image

Steps to reproduce

Image is built through CMake lines 20-38 of the following file https://github.com/ctag-fh-kiel/ctag-tbd/blob/master/CMakeLists.txt

“idf.py build” executes this command (which seems to be correct): C:\msys64\home\rma\ctag-tbd\build && C:\msys64\mingw64\bin\python.exe C:/msys64/home/rma/esp-idf/comp onents/spiffs/spiffsgen.py 0x300000 C:/msys64/home/rma/ctag-tbd/build/spiffs_image C:/msys64/home/rma/ctag-tbd/build/storage.bin --page-size=256 --obj-name-len=32 --meta-len=4 --use-magic --use-magic-len"

which generates a bad spiffs image (storage_bad.bin attached)

Manual execution of $IDF_PATH/components/spiffs/spiffsgen.py 0x300000 build/spiffs_image build/storage.bin sometimes lead to a good image, and sometimes didn’t. It was somehow not reproducible, very strange.

Further information

Original file structure to be put in spiffs image can be found here https://github.com/ctag-fh-kiel/ctag-tbd/tree/master/spiffs_image

I have tried to set up a working esp idf windows environment, as the esp idf installer was not able to successfully create one for me on windows (CMake Version too old in release/v4.1), I have written down my steps here, but ran into above issue (which persists also with a colleague of mine, who uses standard esp idf windows installer): https://github.com/ctag-fh-kiel/ctag-tbd/blob/dev/doc/how-to-dev-windows.md