oss-fuzz: Incorrect coverage report

Sorry, I couldn’t get to this in a while. Also, there was a relevant discussion on llvm-dev mailing list a month ago: https://groups.google.com/d/msg/llvm-dev/Wp0pVlJmhqQ/qT5X_ap7BQAJ

So, what actually collides is probably function’s structural hash mentioned in https://llvm.org/docs/CoverageMappingFormat.html#function-record

It’s also referenced in https://github.com/llvm-mirror/llvm/blob/6b547686c5410b7528212e898fe30fc7ee7a70a3/include/llvm/ProfileData/Coverage/CoverageMapping.h#L714

Which I believe is lower 64 bits of a MD5 hash, but the hash input is the functions profile data rather than a function name.

Haha, nice, thanks for the fix @terrelln! Sorry I was out on Friday. I still want to refresh in my memory what exactly is being hashed, so I’ll leave this issue open for a little while.

The problem is that some function(s) seem to have different binary code in different fuzz targets. That confuses llvm-cov when merging the coverage across binaries (e.g. if the same function has different structure, how we can merge it). The tool prints the warning and aborts the merge process after observing a modified function. If I exclude such cases, everything works fine, e.g.:

warning: /out/dumps/simple_round_trip.profdata: Function basic block count change detected (counter mismatch)

and

warning: /out/dumps/simple_decompress.profdata: Function basic block count change detected (counter mismatch)

without these two targets, I’m getting a correct report.

Usually this happens when different binaries with the same dependencies are compiled with different flags / options.

The most obvious thing to blame in zstd case is ZSTD_STATIC_LINKING_ONLY define which is declared in some targets, but not in all of them. I’ve tried adding #define ZSTD_STATIC_LINKING_ONLY to the following targets as well:

zstd_frame_info.c 
dictionary_round_trip.c 
dictionary_decompress.c 
simple_decompress.c 

root@6342b7c19ccc:/src/zstd/tests/fuzz# egrep ZSTD_STATIC_LINKING_ONLY `egrep -r LLVMFuzzer -l`
zstd_frame_info.c:#define ZSTD_STATIC_LINKING_ONLY
block_round_trip.c:#define ZSTD_STATIC_LINKING_ONLY
stream_round_trip.c:#define ZSTD_STATIC_LINKING_ONLY
block_decompress.c:#define ZSTD_STATIC_LINKING_ONLY
dictionary_round_trip.c:#define ZSTD_STATIC_LINKING_ONLY
simple_round_trip.c:#define ZSTD_STATIC_LINKING_ONLY
dictionary_decompress.c:#define ZSTD_STATIC_LINKING_ONLY
stream_decompress.c:#define ZSTD_STATIC_LINKING_ONLY
simple_decompress.c:#define ZSTD_STATIC_LINKING_ONLY

But still got the error. Do you know if there are any other differences that may lead to the different binary code produced for the same function(s)?

Looks like this has regressed between

https://storage.googleapis.com/oss-fuzz-coverage/zstd/reports/20190411/linux/src/zstd/tests/fuzz/report.html

and

https://storage.googleapis.com/oss-fuzz-coverage/zstd/reports/20190412/linux/src/zstd/tests/fuzz/report.html

i.e. Apr 11-12th. We had a couple changes on that day (#2311 and #2316), but I don’t really suspect either of those is a culprit (though it might be).

Thanks for your report. There is definitely something wrong with the merging of the reports. If I check results for the individual fuzz targets, I see some coverage for all of them, e.g.

https://storage.googleapis.com/oss-fuzz-coverage/zstd/fuzzer_stats/20190417/simple_round_trip.json

shows that simple_round_trip.c has 100% coverage, which is totally reasonable for the target function.

However, this directory clearly lacks coverage for some fuzz targets: https://storage.googleapis.com/oss-fuzz-coverage/zstd/reports/20190417/linux/src/zstd/tests/fuzz/report.html

Sadly, I don’t see any issues reported in the logs. Will have to reproduce it and dig deeper. Thanks again!