cache: zstd compression hanging on Windows with large files
The caching action always hangs when using actions/cache@master on Windows in my test runs using a random 150 MB file.
To dig in more, I created a separate repo that runs the compression commands. On a 1 MB file, it works fine:
2020-05-08T19:01:50.9215932Z bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.5.f-ipp
2020-05-08T19:01:51.7976258Z Create random file
2020-05-08T19:01:51.8560894Z 1+0 records in
2020-05-08T19:01:51.8563149Z 1+0 records out
2020-05-08T19:01:51.8568671Z 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011073 s, 94.7 MB/s
2020-05-08T19:01:51.8578782Z Running zstd standalone
2020-05-08T19:01:52.3317293Z
2020-05-08T19:01:52.3325676Z
2020-05-08T19:01:52.3331145Z Directory: D:\a\test-zstd\test-zstd
2020-05-08T19:01:52.3376822Z
2020-05-08T19:01:52.3392726Z Mode LastWriteTime Length Name
2020-05-08T19:01:52.3395053Z ---- ------------- ------ ----
2020-05-08T19:01:52.3412014Z -a--- 5/8/2020 7:01 PM 1048613 file.zstd
2020-05-08T19:01:52.3412805Z Running tar standalone
2020-05-08T19:01:52.3608654Z -a--- 5/8/2020 7:01 PM 1050112 file.tar
2020-05-08T19:01:52.3622919Z Running tar + zstd in two steps
2020-05-08T19:01:52.4062645Z -a--- 5/8/2020 7:01 PM 1050624 file_combined.tzst
2020-05-08T19:01:52.4064236Z Running tar + zstd using --use-compress-program
2020-05-08T19:01:52.4668289Z -a--- 5/8/2020 7:01 PM 1049135 file.tzst
2020-05-08T19:01:52.4670493Z Done
But on a 150 MB file, it hangs until it I cancel the action:
2020-05-08T19:03:52.6081814Z bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.5.f-ipp
2020-05-08T19:03:53.4979676Z Create random file
2020-05-08T19:03:53.6897822Z 150+0 records in
2020-05-08T19:03:53.6909594Z 150+0 records out
2020-05-08T19:03:53.6912241Z 157286400 bytes (157 MB, 150 MiB) copied, 0.147127 s, 1.1 GB/s
2020-05-08T19:03:53.6925733Z Running zstd standalone
2020-05-08T19:03:54.3540113Z
2020-05-08T19:03:54.3584708Z
2020-05-08T19:03:54.3616876Z Directory: D:\a\test-zstd\test-zstd
2020-05-08T19:03:54.3634999Z
2020-05-08T19:03:54.3648596Z Mode LastWriteTime Length Name
2020-05-08T19:03:54.3650325Z ---- ------------- ------ ----
2020-05-08T19:03:54.3650520Z -a--- 5/8/2020 7:03 PM 157290014 file.zstd
2020-05-08T19:03:54.3650643Z Running tar standalone
2020-05-08T19:03:54.5655674Z -a--- 5/8/2020 7:03 PM 157287936 file.tar
2020-05-08T19:03:54.5658027Z Running tar + zstd in two steps
2020-05-08T19:03:55.0461307Z -a--- 5/8/2020 7:03 PM 157292032 file_combined.tzst
2020-05-08T19:03:55.0498037Z Running tar + zstd using --use-compress-program
2020-05-08T19:06:33.2001585Z ##[error]The operation was canceled.
Test repo - https://github.com/dhadka/test-zstd Example run that’s hanging - https://github.com/dhadka/test-zstd/actions/runs/99414776
CC @imbsky
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 1
- Comments: 27 (14 by maintainers)
Commits related to this issue
- Disable zstd on Windows due to issue #301 — committed to actions/cache by dhadka 4 years ago
- Disable use of cache in Windows for now The Windows cache is using the most of all build caches (over 3GiB), but it is also leas effective as the packaging of the cache takes about 45 min vs. a 55min... — committed to hzeller/verible by hzeller 3 years ago
- Disable use of cache in Windows for now The Windows cache is using the most of all build caches (over 3GiB), but it is also leas effective as the packaging of the cache takes about 45 min vs. a 55min... — committed to hzeller/verible by hzeller 3 years ago
I did a quick test with my workflow to add (a cygwin based, so a bit hacky) gnu tar to PATH so zstd is used on Windows.
So I think it’s worth trying to fix this 😃
I’ve filed a bug with libarchive: https://github.com/libarchive/libarchive/issues/1419
👋🏼 @aminya @lazka @smorimoto We have released a new beta release which should fix this issue. Try tag:
actions/cache@3.1.0-beta. Head over to discussion for feedback: https://github.com/actions/cache/discussions/1019This is a critical issue.
The current behavior is just a cache miss on windows-latest or self-hosted Windows unless zstd.exe is added to the machine.
It’s also a cache miss on any self-runner that doesn’t have zstd, which needs to go into the README. I set up self-hosted machines and had immediate cache failures and it took a lot of debugging to figure out why.
If this has to rely completely on zstd, then it needs to properly fail if zstd isn’t installed.
I will take a look at this tonight if I have enough spare time.
Thanks @lazka 🙂
@Cyan4973 I suspect it’s an issue with bsdtar. I tested with a different compression tool (https://blog.kowalczyk.info/software/pigz-for-windows.html) and it also hangs on large files.