scraper: Memory corruption (probably since 0.21.0)

We have a service at my company that uses scraper for extracting data from HTML that recently started crashing. I’ve tracked down the issue that appears after bumping scraper from 0.19.0 to the latest version.

The crash happens once in a blue moon with the system allocator, however with Jemalloc on linux, it happens almost immediately.

I have extracted some of our logic that consistently reproduces the issue and uploaded it here: https://github.com/tamasfe/scraper-segfault . Simply run cargo run and observe the segmentation fault message.

I did not yet dive into the causes and attempts to fix it as tracking it down took me several hours already, I hope the repro is minimal enough for people who are familiar with the project internals.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 3
  • Comments: 26 (21 by maintainers)

Most upvoted comments

I used this crate v0.20.0 with wasm-bindgen, which used to compile and run just fine. But since v0.21.0, there seems to be some issue with allocator. With console error hook, I identified that the dlmalloc code panics.

panicked at /rust/deps/dlmalloc-0.2.7/src/dlmalloc.rs:1198:13:
assertion failed: psize <= size + max_overhead

I suspect that this issue is related to my wasm issue, and might be related to https://github.com/rustwasm/wasm-pack/issues/1389. (I’m on cargo 1.86.0 btw)

Also I’ve noticed that I was using Html::parse_fragment, changing it to Html::parse_document works correctly, which is fair since we are parsing an entire document, I don’t know what difference that makes though.

Sorry, it works on the reduced test data, but running it with the original test data, it triggers a panic, which may or may not be related to this:

thread 'main' panicked at scraper/scraper/src/html/tree_sink.rs:64:18:
called `Option::unwrap()` on a `None` value

Running the initial reproduction but using parse_document repeatedly, I sometimes get just a segfault and sometimes this panic with a follow-up segfault. As written above, this really smells of stack-based memory corruption which of course makes the backtraces and such dubious.

Oddly enough the bug does not seem to trigger it.

I get a panic in our tree sink implementation, i.e. html5ever wants to know the name of an element which does not exist. And then Rust’s panic handler segfaults trying to create a backtrace which suggests stack-based memory corruption (which jives with Valgrind’s report).

miri over the weekend

I tried Miri, but stacked borrows fails somewhere inside tendril and tree borrows does not handle the pointer-to-integer casts in string-cache.