Efficient-Compression-Tool: Infinite loop via FileOptimizer since 0.9.2
I’m a big user of FileOptimizer that calls up ECT as part of a chain.
So lately (as of version 16 of FileOptimizer) ECT has been going into an infite loop at random, usually after processing 10-15 files (by infinite loop I mean ECT is running, using computer cycles and yet never seems to complete it’s task).
As per the joined screenshot, when in this loop ECT hovers at one thread and 11Mb of ram. I’ve tried to kill processes and liberate ram to no avail.
I’ve also reproduced the issue via a batchfile, called via a console with admin rights, using the following command line :
FOR /f "tokens=*" %%G IN ('dir *.PNG /b') DO (ECT.exe --mt-deflate --mt-file --allfilters -strip -90032 "%%G" >> ECT.LOG)
With the batch file it did take longer to hit a snag (122 files) but it did choke again.
This is a wild guess but I’m thinking here ECT is dynamically reducing the ressources to adapt to whatever other programs are running simultanously but seems to have an issue clawing back the ressources when they are available again. Although I have plenty ram and threads to go when the issue happens.
This might also be CPU-specific.
I can send you a pack with 150 of the PNGs I was using during this test so you can compare apples to apples.
Tried with ECT 0.9.4 and the same behavior is present.
I’m unable to go any further since ECT doesn’t spit out an error message. It just runs forever.
The logfile doesn’t yield any further information.
I’m more than willing to give you more information if needed.
Here are my specs: AMD Ryzen 5 2400G with Radeon Vega Graphics (integrated graphics disabled because of PCI-e video card) 4GB NVIDIA GeForce GTX 1650 SUPER, latest gaming drivers TUF B450M-PLUS GAMING (Latest BIOS) () Windows 10 pro 64 bits (French) (11.789.19041.0) 32,0 Go Dual-Channel (tested with Memtest86+) (Re-size BAR enabled) System is rock-solid so it’s not a BIOS configuration issue / bad ram.
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 44 (16 by maintainers)
I investigated this using LLDB and found the underlying cause – this has been resolved in ec10745. The infinite loop could happen when ECT uses a fast path designed for long runs of repeated bytes. In the fast path the number of repeated bytes is counted and the repeated section is skipped to avoid updating the match finder at each position. The bug was that the match finder was not re-initialized properly afterwards, causing an infinite loop based on the binary structure being corrupted. I have published a new release (version 0.9.5) that contains the bug fix.
Thank you all for your input in investigating this! Feel free to reopen the issue if you still experience infinite loops, although they can sometimes be confused with ECT just running slowly when using a large number of iterations or aggressive filter options. The change above appears to fix the issue on my side, but it is difficult to rule out that there are still related problems due to its non-deterministic nature.
I had also the issue with an Intel. So it’s not an processor related issue.
I had the problem with both Intel and ARM Mac, so I don’t think that makes a difference.
The problem is likely in the program logic itself – changing permissions or running in a VM will not affect this.
Since the given changes do not solve the problem, it’ll be difficult for me to track it down without some local debugging – if you are comfortable sharing the data set reproducing it locally within GDB should reveal where the issue occurs, otherwise I could run the problem with asan/ubsan for longer periods. With that being said, I am relatively busy with other things currently, it might take a while for me to test and try to reproduce the issue.
Created a binary that disables
GetMatches2()and related changes, I can’t point to any specific cause why it would be responsible but it looks like the most likely candidate. ect.exe.gz Code is available on the freeze-debug branch. Let me know if you are still having issues with it, if not it would really narrow down where the issue is happening.I did some testing with Asan and UBsan but wasn’t able to find anything immediately suspicious. Looking at the code itself 89d5622 is perhaps the most likely change set that could have introduced issues. I’ll look at it in-depth later on, if I have found a potential cause I’ll try making a new binary and you can test if that fixes the issues on your data set.
I’ll take a look in the coming days, but this might still be hard to track down. Knowing that it happens starting with 0.9.2 is certainly useful though.
Sorry, can’t reproduce my earlier issues. I clearly remember ECT 0.9.3 being stuck on the files for 20+ days without even reaching genetic filtering, but now both 0.9.3 and 0.9.4 complete the files within half an hour. Please disregard my report.
FWIW, I noticed ECT hanging on two PNGs of mine as well, and this problem surfaced last summer. I too use high optimization settings with multiple block-splitting passes (
-30060).I found one of the affected PNG and I’m trying to reproduce the hang right now with the latest build. If successful, I will attach the image tomorrow.