godot: 3.2.2 Linux headless version exports invalid Mac app

Actually it’s just constant size issue, changed it to:

zipfi.external_fa = (uint32_t)(is_executable ? 0100755 : 0100644) << 16L;

(results in following output by zip-info, and is extracted correctly by all apps I have tested):

  Unix file attributes (100755 octal):            -rwxr-xr-x
  MS-DOS file attributes (00 hex):                none

instead of

zipfi.external_fa = (is_executable ? 0100755 : 0100644) << 16L;

(results in following output by zip-info, and have broken files when extracted by Archive Util)

  Unix file attributes (177777 octal):            ?rwsrwsrwt
  MS-DOS file attributes (FF hex):                rdo hid sys lab dir arc lnk exe

or

zipfi.external_fa = (is_executable ? 0755 : 0644) << 16L;

(results in following output by zip-info, and seems to be extracted correctly)

  Unix file attributes (000755 octal):            ?rwxr-xr-x
  MS-DOS file attributes (00 hex):                none

Also, I have checked info-zip source and it’s setting attributes as following (result seems to be the same, not sure why it’s done it this way):

uint32_t _mode = (is_executable ? 0100755 : 0100644);

zipfi.external_fa = (_mode << 16L) | !(_mode & 0200);
zipfi.internal_fa = 0;

Thanks @firebelley! That means this regression is due to my changes in #39700.

I don’t really understand why, but it seems the file type flags I’ve added are invalid when zipping from a linux system then. A possible hotfix might be to set zip flags differently depending on the platform we’re exporting from. I’ll try and reproduce it in the coming days to see if I can figure it out.

@bruvzg I agree it would be worth checking with a different zip library, although at this point I have no idea if the problems we’re encountering are mostly coming from the library itself or some weirdness in macOS archivers 😃

Skimming this thread, am I correct to assume that we’ll have to wait for 4.0 until headless export to OSX works, and will have to export using the normal editor build in the meantime?

Good catch! That makes a lot of sense. Bit-wise operators on signed integers are undefined and probably made the result compiler specific.

It’s strange. I’m not getting the same results as you do, and my change is only setting some standard “normal file” flag on all files 😕 I agree it makes sense to revert #39700 for now though, and if needed we’ll make another fix later.

The next thing I’ll do is testing again with the official 3.2.2 release to see if my problem could be specific to my compilation of godot somehow.

Here’s my previous test case scenario:

Zip export created on Windows 10 (godot master, compiled with vs 2019) Zip extracted on OSX, version: 10.13.6 (High Sierra)

Before #39700: Doesn’t work with default Archive Utility (file not executable) Works when extracted using unzip in command line

After #39700: Works with both Archive Utility & unzip in command line

Given the confusion that’s resulted from https://github.com/godotengine/godot/issues/527 still being open despite https://github.com/godotengine/godot/pull/33447 & https://github.com/godotengine/godot/issues/527#issuecomment-500594602, would it be good to:

1. Close #527 (Assuming that pre this regression, Linux+Windows export to Mac did work).
2. Re-title this issue to be “3.2.2 exports invalid Mac app .zip on all platforms”.
3. Revert #39700 and then close this? (Ideally with a hotfix release?)
4. Create new dedicated issue(s) for any other outstanding variations on this issue such as that mentioned by @pouleyKetchoupp?

BTW @pouleyKetchoupp during my research I discovered this project which seems to document some of the issues they encountered with Mac + Zip support:

• https://github.com/thejoshwolfe/yazl#regarding-zip64-support
• https://github.com/thejoshwolfe/yazl#output-structure

(It was described in https://github.com/sindresorhus/gulp-zip/issues/38#issuecomment-75138908 as “…spent about 6 hours today working with every zip library for node. [It was] The only one that has worked…” )

Some extra info:

• 3.2.1 - separate code path for DMG and ZIP, ZIP export copy files along with attributes from the template ZIP, this was not working in some cases for unknown reasons and result in non-executable files (probably issues with the template ZIP permissions, but there were multiple issues on Linux and Windows popping up constantly).
• 3.2.2 - (before #39700) export was changed to use same code path for both DMG and ZIP (extract files from template ZIP to temp folder, sign fully formed .app and compress to new ZIP), primary to support signing, ZIP fs type in header was changed form default to Unix, since default one was not preserving executable flag (specific combination of header flag and ext file attributes was found experimentally, to work with at least Archive util, Keka and macOS default command line unzip (infozip)).

Also checked it on the current 3.2 head, same effect, files full of 0x00 when extracted with Archive Util, normal when extracting with Keka. Reverting #39700 do fix it and extracted files have correct permissions with both Archive Util and Keka.

If it helps, here is the output app: https://drive.google.com/file/d/1-HJnBxJ3-n8XokI-NBsz0QA3xBncPL7P/view?usp=sharing

I have rechecked this ZIP and it seems to have +x on executable when extracting with both Archive Utility and Keka, but result of Archive Utility contains only 0x00 in all files (size is valid)!