zlib-ng: pigz compiled with zlib-ng generates errors with level=1

@reddydp discovered that pigz compiled with zlib-ng generates errors with compression level 1. This is specific to zlib-ng, as versions of pigz compiled to default, Intel or CloudFlare zlib does not show this error. Since the zlib-ng and lib-Intel use similar strategy for level-1 compression, I wonder if this is related to a fixed intel zlib issue.

We have replicated the issue on Linux as well as MacOS. To see the issue run the following commands:

 git clone git clone https://github.com/neurolabusc/pigz-bench
 cd pigz-bench
 ./1compile.sh 
 ./4verify.sh

The issue is only seen with pigzNG with the level 1 compression:

Segmentation fault: 11  $cmd
pigzNG	1	1	33
pigzNG: skipping: /Users/chris/src/pigz-bench/corpus/asl16.nii.gz: corrupted -- invalid deflate data (invalid stored block lengths)

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 32 (20 by maintainers)

Commits related to this issue

Enable Level 1 testing (https://github.com/zlib-ng/zlib-ng/issues/536) — committed to neurolabusc/pigz-bench-python by neurolabusc 4 years ago
Turn off compare in GH-536 tests. We can't compare against original because we are using switchlevels and only compressing certain parts of the original. — committed to nmoinvaz/zlib-ng by nmoinvaz 4 years ago
Turn off compare and gzip verify in GH-536 tests due to switchlevels. — committed to nmoinvaz/zlib-ng by nmoinvaz 4 years ago
Turn off compare and gzip verify in GH-536 tests due to switchlevels. — committed to nmoinvaz/zlib-ng by nmoinvaz 4 years ago
Turn off compare and gzip verify in GH-536 tests due to switchlevels. — committed to nmoinvaz/zlib-ng by nmoinvaz 4 years ago
Turn off compare and gzip verify in GH-536 tests due to switchlevels. — committed to nmoinvaz/zlib-ng by nmoinvaz 4 years ago
Turn off compare and gzip verify in GH-536 tests due to switchlevels. — committed to zlib-ng/zlib-ng by nmoinvaz 4 years ago

Most upvoted comments

Your recent commit works on all my computers with both the Silesia corpus and my neuroimaging data. If this can be merged as a pull request, I think this will close the issue. Thanks!

Below are the results for the 3900X

DecompressMethod	ms	mb/s
gzip	11045	186.49
pigzCloudflare	6378	322.92
pigzSystem	6388	322.41
pigzNM	5276	390.39

Silesia Compression: silesia_speed_size Neuroimaging Compression: corpus_speed_size

neurolabusc on Jun 3, 2020

Should be fixed now in latest develop.

nmoinvaz on Jun 19, 2020

It appears with Z_BLOCK it shouldn’t write a block unless there is s->strm->avail_in. This the reason for the s->sym_next check at the bottom of deflate_fast. @neurolabusc, can you test against my tests/def-v1 branch and see if the error happens?

nmoinvaz on Jun 2, 2020