slatedb: GC deletes things it shouldn't
Describe the bug GC deletes compacted ssts when it shouldn’t, we end up with a corrupted database/manifest.
To Reproduce
- Set short gc values
Such as:
db_options.garbage_collector_options = Some(GarbageCollectorOptions {
wal_options: Some(GarbageCollectorDirectoryOptions {
interval: Some(Duration::from_secs(60)),
min_age: Duration::from_secs(60),
}),
manifest_options: Some(GarbageCollectorDirectoryOptions {
interval: Some(Duration::from_secs(60)),
min_age: Duration::from_secs(60),
}),
compacted_options: Some(GarbageCollectorDirectoryOptions {
interval: Some(Duration::from_secs(60)),
min_age: Duration::from_secs(60),
}),
..default::Default::default()
});
- Ingest data at a high rate.
Error: "Storage error: SlateDB error: Object store error: Object at location /mnt/ct_storage/ct_log/compacted/01JX8DGH1K0PXJ4KS8DHPNAPVM.sst not found: No such file or directory (os error 2)"
trace
2025-06-08T20:19:06.459780Z DEBUG slatedb::garbage_collector: Scheduled garbage collection attempt for Compacted SSTs.
2025-06-08T20:19:06.463024Z DEBUG slatedb::garbage_collector: GC has collected 0 Manifests, 0 WAL SSTs and 0 Compacted SSTs.
thread 'tokio-runtime-worker' panicked at src/api/handlers.rs:477:24:
called `Result::unwrap()` on an `Err` value: InvalidFormat("SlateDB error during batch processing")
stack backtrace:
0: 0x5556a33319d2 - std::backtrace_rs::backtrace::libunwind::trace::hc20b48b31ee52608
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
1: 0x5556a33319d2 - std::backtrace_rs::backtrace::trace_unsynchronized::h5d207cd20f193d88
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
2: 0x5556a33319d2 - std::sys::backtrace::_print_fmt::h46402c32145e026c
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:66:9
3: 0x5556a33319d2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc04c8f544ab24d66
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:39:26
4: 0x5556a335e813 - core::fmt::rt::Argument::fmt::h2bcd587ea50ac9a4
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/fmt/rt.rs:184:76
5: 0x5556a335e813 - core::fmt::write::hfe57b7174b7d8eab
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/fmt/mod.rs:1481:21
6: 0x5556a332dc73 - std::io::default_write_fmt::h20c7e2d50577e95e
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/io/mod.rs:639:11
7: 0x5556a332dc73 - std::io::Write::write_fmt::h154385efa8565236
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/io/mod.rs:1914:13
8: 0x5556a3331822 - std::sys::backtrace::BacktraceLock::print::h0c8f24e22f5873a8
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:42:9
9: 0x5556a33326d2 - std::panicking::default_hook::{{closure}}::hd07d57e6a602c8e4
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:300:22
10: 0x5556a33324d5 - std::panicking::default_hook::h63d12f7d95bd91ed
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:327:9
11: 0x5556a3333072 - std::panicking::rust_panic_with_hook::h33b18b24045abff4
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:833:13
12: 0x5556a3332e1a - std::panicking::begin_panic_handler::{{closure}}::hf8313cc2fd0126bc
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:706:13
13: 0x5556a3331ed9 - std::sys::backtrace::__rust_end_short_backtrace::h57fe07c8aea5c98a
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:168:18
14: 0x5556a3332aad - __rustc[95feac21a9532783]::rust_begin_unwind
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
15: 0x5556a2a90170 - core::panicking::panic_fmt::hd54fb667be51beea
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
16: 0x5556a2a90596 - core::result::unwrap_failed::he8e27e02739cd3d2
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1704:5
17: 0x5556a2c6c9ed - core::result::Result<T,E>::unwrap::he7c2f1b26fe28005
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1109:23
18: 0x5556a2c6c9ed - compactlog::api::handlers::get_entries::{{closure}}::h6d98f7613e594e3c
at project/src/api/handlers.rs:477:9
19: 0x5556a2c6c9ed - <F as axum::handler::Handler<(M,T1,T2),S>>::call::{{closure}}::hc71f266487642d97
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/handler/mod.rs:241:53
20: 0x5556a2b5dbd1 - <core::pin::Pin<P> as core::future::future::Future>::poll::h37aab4f28224015e
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
21: 0x5556a2b5dbd1 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h7e164de95090df14
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:37
22: 0x5556a2b5dd7e - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::h0c03d67dc2064131
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
23: 0x5556a2b5dd7e - <axum::handler::future::IntoServiceFuture<F> as core::future::future::Future>::poll::habf5f31815c63f2c
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/macros.rs:42:17
24: 0x5556a2b5dd7e - <F as futures_core::future::TryFuture>::try_poll::h9aa6827982a015d6
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92:9
25: 0x5556a2b5dd7e - <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll::h37af497a50fc4662
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34:9
26: 0x5556a2b5dd7e - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hc47b70b1ac90d241
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:37
27: 0x5556a2c77a69 - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::hd82613c5f7434699
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
28: 0x5556a2c77a69 - <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll::h689d03630ec6f7ef
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
29: 0x5556a2c77a69 - <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll::h284c177bfac51f1e
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-0.5.2/src/macros.rs:38:17
30: 0x5556a2c4d7de - <core::pin::Pin<P> as core::future::future::Future>::poll::h89938290833e0ae9
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
31: 0x5556a2c4d7de - <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h853d89f71838f077
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-0.5.2/src/util/oneshot.rs:97:38
32: 0x5556a2b5d464 - <axum::routing::route::RouteFuture<E> as core::future::future::Future>::poll::h091c35c4edb6a6f8
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/routing/route.rs:182:61
33: 0x5556a2b5d464 - <hyper_util::service::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h2b5aa78ffa287a17
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-util-0.1.13/src/service/oneshot.rs:55:38
34: 0x5556a2c9344f - <hyper_util::service::glue::TowerToHyperServiceFuture<S,R> as core::future::future::Future>::poll::h96826e6ddfc20c27
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-util-0.1.13/src/service/glue.rs:70:9
35: 0x5556a2c9344f - <hyper::proto::h1::dispatch::Server<S,hyper::body::incoming::Incoming> as hyper::proto::h1::dispatch::Dispatch>::poll_msg::hc7ddc9d55f23e920
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:538:35
36: 0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_write::head51ebca3732f4b
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:336:43
37: 0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_loop::h65f9d22b74946fc9
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:173:21
38: 0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_inner::h37b0785ce677d5c4
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:149:16
39: 0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_catch::h7b1be398515b205c
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:128:28
40: 0x5556a2b5fcf4 - <hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T> as core::future::future::Future>::poll::hc7ae0e3611d81f6d
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:471:9
41: 0x5556a2b5fcf4 - <hyper::server::conn::http1::UpgradeableConnection<I,S> as core::future::future::Future>::poll::hfbee6c45f04cc158
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/server/conn/http1.rs:530:51
42: 0x5556a2b0f719 - <hyper_util::server::conn::auto::UpgradeableConnection<I,S,E> as core::future::future::Future>::poll::h07938d6713295559
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-util-0.1.13/src/server/conn/auto/mod.rs:645:28
43: 0x5556a2bf4fee - <axum::serve::Serve<M,S> as core::future::into_future::IntoFuture>::into_future::{{closure}}::{{closure}}::h93f4ee28963bf63e
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/serve.rs:257:26
44: 0x5556a2bf2217 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h861cb07729a876aa
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:331:17
45: 0x5556a2bf2217 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h81c98e46e790608c
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/loom/std/unsafe_cell.rs:16:9
46: 0x5556a2bf2217 - tokio::runtime::task::core::Core<T,S>::poll::h38841df5f616cca9
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:320:30
47: 0x5556a2c4901f - tokio::runtime::task::harness::poll_future::{{closure}}::h660f5e872383dcc8
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:532:19
48: 0x5556a2c4901f - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h317aacec921589f8
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
49: 0x5556a2c4901f - std::panicking::try::do_call::hc27b14cf34a6773e
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
50: 0x5556a2c4901f - std::panicking::try::h8130c75a669905b3
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
51: 0x5556a2c4901f - std::panic::catch_unwind::h93019168ce90aebc
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
52: 0x5556a2c4901f - tokio::runtime::task::harness::poll_future::h5ead0b51ae04c6f9
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:520:18
53: 0x5556a2c4901f - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h30f13a22c00db7dd
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:209:27
54: 0x5556a2c4901f - tokio::runtime::task::harness::Harness<T,S>::poll::h6473ac62d3b781bc
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:154:15
55: 0x5556a2c4901f - tokio::runtime::task::raw::poll::h4b2fb2e6df391161
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:271:5
56: 0x5556a31f71f3 - tokio::runtime::task::raw::RawTask::poll::h80a2b40cbfc6d628
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:201:18
57: 0x5556a31f71f3 - tokio::runtime::task::LocalNotified<S>::run::h0265f707d73c64d9
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/mod.rs:463:9
58: 0x5556a31f71f3 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h402655e5ffaa975e
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:600:18
59: 0x5556a31f71f3 - tokio::task::coop::with_budget::h64d1d3518588bab2
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:167:5
60: 0x5556a31f71f3 - tokio::task::coop::budget::h57c0ef804492b8ae
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:133:5
61: 0x5556a31f71f3 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h97d0560d2d3bffeb
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:594:9
62: 0x5556a31f66b0 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h1cd264141ff5c42a
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:542:24
63: 0x5556a31f0d18 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h9c629e6550bf7295
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:507:21
64: 0x5556a31f0d18 - tokio::runtime::context::scoped::Scoped<T>::set::hc73416f9295f3423
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/scoped.rs:40:9
65: 0x5556a3200c68 - tokio::runtime::context::set_scheduler::{{closure}}::hf3cd1df9d97c5171
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:26
66: 0x5556a3200c68 - std::thread::local::LocalKey<T>::try_with::h787240947063199e
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:311:12
67: 0x5556a3200c68 - std::thread::local::LocalKey<T>::with::h6c2f0afda759747b
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:275:15
68: 0x5556a3200c68 - tokio::runtime::context::set_scheduler::h726f0c4df2c776eb
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:9
69: 0x5556a3200c68 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h044afcd2983ac2e9
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:502:9
70: 0x5556a3200c68 - tokio::runtime::context::runtime::enter_runtime::h671b7106621a36c4
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/runtime.rs:65:16
71: 0x5556a31f5f2d - tokio::runtime::scheduler::multi_thread::worker::run::hf6aeb371ff986937
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:494:5
72: 0x5556a31ecf59 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h400ccdd1521a3aff
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:460:45
73: 0x5556a31ecf59 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h081956de72a1e89e
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/task.rs:42:21
74: 0x5556a31ecf59 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h47ef6b64ded94dd7
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:331:17
75: 0x5556a31ecf59 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb960060d1fad196f
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/loom/std/unsafe_cell.rs:16:9
76: 0x5556a31ecf59 - tokio::runtime::task::core::Core<T,S>::poll::h94131ad7d3e48ca3
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:320:30
77: 0x5556a31e77b1 - tokio::runtime::task::harness::poll_future::{{closure}}::hc08335a4185f9c18
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:532:19
78: 0x5556a31e77b1 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h18a1c4cba872391d
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
79: 0x5556a31e77b1 - std::panicking::try::do_call::h02a74b2f1b882343
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
80: 0x5556a31e77b1 - std::panicking::try::h261aa36c47cfaed5
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
81: 0x5556a31e77b1 - std::panic::catch_unwind::heaea8a63fdc403f4
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
82: 0x5556a31e77b1 - tokio::runtime::task::harness::poll_future::hc4d0e2d2185685a5
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:520:18
83: 0x5556a31e77b1 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hcfb3ecb006131488
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:209:27
84: 0x5556a31e77b1 - tokio::runtime::task::harness::Harness<T,S>::poll::h43dcca0c1d5fe707
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:154:15
85: 0x5556a320b657 - tokio::runtime::task::raw::RawTask::poll::h80a2b40cbfc6d628
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:201:18
86: 0x5556a320b657 - tokio::runtime::task::UnownedTask<S>::run::h9ed99ee43f82ff54
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/mod.rs:500:9
87: 0x5556a320b657 - tokio::runtime::blocking::pool::Task::run::he5db632505af0aa5
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/pool.rs:161:9
88: 0x5556a320b657 - tokio::runtime::blocking::pool::Inner::run::h8ae30c679d447f9b
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/pool.rs:511:17
89: 0x5556a31f7c40 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h4909d75f996fb58b
at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/pool.rs:469:13
90: 0x5556a31f7c40 - std::sys::backtrace::__rust_begin_short_backtrace::h1a39e32b38e38362
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:152:18
91: 0x5556a31f8688 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h4a9daaa72133bd41
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/mod.rs:559:17
92: 0x5556a31f8688 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h9a9f5fb0a4c36a36
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
93: 0x5556a31f8688 - std::panicking::try::do_call::h2e908e64532c5902
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
94: 0x5556a31f8688 - std::panicking::try::he17cf4bdba03423d
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
95: 0x5556a31f8688 - std::panic::catch_unwind::h813ba417dc2c2072
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
96: 0x5556a31f8688 - std::thread::Builder::spawn_unchecked_::{{closure}}::h39d1083231ffe237
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/mod.rs:557:30
97: 0x5556a31f8688 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h5642d3d0eb327fd1
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
98: 0x5556a333507b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc031c115ad99256b
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1966:9
99: 0x5556a333507b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha14f846e5cecd1a7
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1966:9
100: 0x5556a333507b - std::sys::pal::unix::thread::Thread::new::thread_start::h2717d4f9aca577e6
at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/pal/unix/thread.rs:109:17
101: 0x7f7b238c21f5 - <unknown>
102: 0x7f7b2394289c - <unknown>
103: 0x0 - <unknown>```
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 20 (14 by maintainers)
We are planning to support compactors, garbage collectors, readers, and the writer each running on a separate machine. Unfortunately, that prevents us from this type of coordination. This is why Rohan was suggesting modifying the manifest.
@rodesai Thinking a bit more about implementation. Regarding the manifest protocol, what do you think about using UUID v7 (time sortable) UUIDs for L0+ and then maintaining a compacted low watermark? The compactor can update the low watermark as it compacts, and the GC isn’t allowed to delete anything > the low watermark UUID (even if it passes min_age).
@criccomini I posted a repro repo just above (just make sure you are on the corruption branch), that should just “work” to produce the same thing after few mins
That shouldn’t be the case. The GC as its implemented looks at all manifests referenced by a checkpoint.
list_active_l0_and_compacted_sstscallsmanifest_store.read_active_manifestsand builds a list of ssts from all the manifests that returens. read_active_manifests should look through all the checkpoints.There’s a test for this case here: https://github.com/slatedb/slatedb/blob/main/slatedb/src/garbage_collector.rs#L913