honggfuzz: Some fuzz targets instrumented with honggfuzz seem to crash sometimes

I haven’t figured out what’s going on yet but it appears some fuzz targets segfault from time to time when they are run with honggfuzz and don’t crash when they are run directly. The verifier can’t reproduce those crashes either

Crash: saved as '/home/vagrant/systemd/SIGSEGV.PC.7fffe517d7d7.STACK.18f57b0756.CODE.1.ADDR.7fffff7feff8.INSTR.call___0xfffffffffff0d8d9.fuzz'
Sz:4105 Tm:27,835,441us (i/b/h/e/p/c) New:0/0/0/0/0/100, Cur:0/0/0/1/0/2326
Launching verifier for HASH: 18f57b0756 (iteration: 1 out of 5)
[2022-06-08T20:07:05+0000][E][2048098] fuzz_runVerifier():280 Verifier stack mismatch: (original) 18f57b0756 != (new) 0

I’m not sure if it helps but backtraces look like

PID 2046163 - core
TID 2046163:
#0  0x00007fffe517d7b2 dl_iterate_phdr
#1  0x00007fffeeea7239 __wrap_memcmp
#2  0x0000000000432a44 instrumentAddConstMem
#3  0x00007fffeeea72b8 __wrap_memcmp
#4  0x0000000000432ab6 instrumentAddConstMem
#5  0x00007fffeeea72b8 __wrap_memcmp
#6  0x0000000000432a44 instrumentAddConstMem
#7  0x00007fffeeea72b8 __wrap_memcmp
#8  0x0000000000432ad4 instrumentAddConstMem
#9  0x00007fffeeea72b8 __wrap_memcmp
#10 0x0000000000432a44 instrumentAddConstMem
#11 0x00007fffeeea72b8 __wrap_memcmp
...
#116382 0x0000000000432ab6 instrumentAddConstMem
#116383 0x00007fffeeea72b8 __wrap_memcmp
#116384 0x0000000000432ab6 instrumentAddConstMem
#116385 0x00007fffeeea72b8 __wrap_memcmp
#116386 0x0000000000432ad4 instrumentAddConstMem
#116387 0x00007fffeeea72b8 __wrap_memcmp
#116388 0x0000000000432b1e instrumentAddConstMem
#116389 0x00007fffeeea72b8 __wrap_memcmp
#116390 0x000000000043343d instrumentAddConstStrN
#116391 0x0000000000433ee6 __wrap_strncmp
#116392 0x00007fffe57c8f68 startswith
#116393 0x00007fffe567a5a9 documentation_url_is_valid
#116394 0x00007fffeed24013 config_parse_documentation
#116395 0x00007fffe5435569 next_assignment
#116396 0x00007fffe542688f parse_line
#116397 0x00007fffe5424fee config_parse
#116398 0x000000000042f0e7 LLVMFuzzerTestOneInput
#116399 0x000000000042fb13 main
#116400 0x00007fffe5028440 __libc_start_call_main
#116401 0x00007fffe50284f0 __libc_start_main@@GLIBC_2.34
#116402 0x0000000000406785 _start

That particular fuzz target was built with hfuzz-clang/hfuzz-clang++ on Fedora 35 with clang version 13.0.0 (Fedora 13.0.0-3.fc35) with no sanitizers.

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 24 (11 by maintainers)

Commits related to this issue

infra: let honggfuzz verify crashes to make it easier to catch issues like https://github.com/google/honggfuzz/issues/465. With -V when unreproducible crashes are triggered honggfuzz shows something... — committed to evverx/oss-fuzz by evverx 2 years ago
infra: let honggfuzz verify crashes (#7850) to make it easier to catch issues like https://github.com/google/honggfuzz/issues/465. With -V when unreproducible crashes are triggered honggfuzz show... — committed to google/oss-fuzz by evverx 2 years ago
infra: let honggfuzz verify crashes (#7850) to make it easier to catch issues like https://github.com/google/honggfuzz/issues/465. With -V when unreproducible crashes are triggered honggfuzz show... — committed to MartinPetkov/oss-fuzz by evverx 2 years ago

Most upvoted comments

The problem was interesting. When two (or more) instrumented by honggfuzz modules (binaries, libraries) end up in one address space, then libc_memcmp will be resolved to the other’s module’s memcmp (which will in turn call into the original one).

I fixed it temporarily with - https://github.com/google/honggfuzz/commit/a338c5322dea64c468a4419d00daebd635f7ac8b - but I need to think how to do it “for good”, as the fix is not pretty.

I updated the oss-fuzz tag too.

robertswiecki on Jun 23, 2022

The updated honggfuzz didn’t break anything. There were test failures but all due to sourceware.org being down.

jonathanmetzman on Jun 16, 2022