regex: calling `caputres_iter` in an `async` function may cause memory leak
What version of regex are you using?
Latest 1.10.5
Describe the bug at a high level.
Calling the static regular instance’s caputres_iter in an async function may cause memory leak.
What are the steps to reproduce the behavior?
// complete code: https://github.com/ziyoung/memory-leak-example/blob/master/src/main.rs
static REG: Lazy<Regex> = Lazy::new(|| Regex::new(r#"(Object\.\w+)"#).unwrap());
async fn memory_leak_demo() -> Vec<u8> {
let text = reqwest::get("https://cdnjs.cloudflare.com/ajax/libs/vue/3.4.34/vue.cjs.js")
.await
.unwrap()
.text()
.await
.unwrap();
let mut buf = Vec::from(text.as_bytes());
for (i, caps) in REG.captures_iter(&text).enumerate() {
println!("process text: {}", i + 1);
tokio::time::sleep(Duration::from_millis(50)).await;
let m = caps.get(0).unwrap();
let start = m.start();
let end = m.end();
let text = &text[start..end];
buf.extend(text.as_bytes());
}
buf
}
async fn task() {
for _ in 0..50 {
let buf = memory_leak_demo().await;
let s = String::from_utf8(buf).unwrap_or_default();
println!("string length: {}", s.len());
}
}
#[tokio::main]
async fn main() {
tokio::spawn(async {
for i in 0..5 {
task().await;
}
})
.await
.unwrap();
println!("done");
}
What is the actual behavior?
The memory allocated during the execution of the captures_iter method was not released. I dump the memory profile file during the program running. https://github.com/ziyoung/memory-leak-example?tab=readme-ov-file#regex-memory-issue
If I change my code like follwings, there’s no memory leak. https://github.com/ziyoung/memory-leak-example/pull/1/files#diff-42cb6807ad74b3e201c5a7ca98b911c5fa08380e942be6e4ac5807f8377f87fc
+ fn get_tag_positions(src: &str) -> Vec<(usize, usize)> {
+ REG
+ .clone()
+ .captures_iter(src)
+ .map(|caps| {
+ let m = caps.get(0).unwrap();
+ (m.start(), m.end())
+ })
+ .collect::<Vec<_>>()
+ }
// ...
- for (i, caps) in REG.captures_iter(&text).enumerate() {
+ for (i, (start, end)) in get_tag_positions(&text).into_iter().enumerate() {
What is the expected behavior?
There’s no memory leak.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20 (10 by maintainers)
Ah I see. That’s interesting. I do still wonder if number of threads has a role to play here though, since each cache line is a stack of reusable allocations. As long as there isn’t any contention, these stacks can continue to grow I think. But I suppose growth is only possible when there is actual concurrency happening.
Some thoughts:
regex, I would prefer not to, of if so, something simplified. Forregex-automata, I’m generally more open to exposing more knobs.This doesn’t imply a leak. When you run a regex, it has to allocate space to run the search. This space is tied to the
Regexand it’s cached in a thread safe manner. The amount of memory allocated scales proportionally to the maximum number of simultaneous requests. Not all memory allocated is kept, but some is.When you clone the
Regex, you’re creating a new pool of memory that gets dropped at the end of the function.The expectation that all memory allocated during a search gets released is just wrong. A memory leak is something that happens when memory used grows without bound given an otherwise fixed set of resources.