slatedb: GC deletes things it shouldn't

Describe the bug GC deletes compacted ssts when it shouldn’t, we end up with a corrupted database/manifest.

To Reproduce

  1. Set short gc values

Such as:


db_options.garbage_collector_options = Some(GarbageCollectorOptions {
        wal_options: Some(GarbageCollectorDirectoryOptions {
            interval: Some(Duration::from_secs(60)),
            min_age: Duration::from_secs(60),
        }),
        manifest_options: Some(GarbageCollectorDirectoryOptions {
            interval: Some(Duration::from_secs(60)),
            min_age: Duration::from_secs(60),
        }),
        compacted_options: Some(GarbageCollectorDirectoryOptions {
            interval: Some(Duration::from_secs(60)),
            min_age: Duration::from_secs(60),
        }),
        ..default::Default::default()
    });
  1. Ingest data at a high rate.

Error: "Storage error: SlateDB error: Object store error: Object at location /mnt/ct_storage/ct_log/compacted/01JX8DGH1K0PXJ4KS8DHPNAPVM.sst not found: No such file or directory (os error 2)"

trace

2025-06-08T20:19:06.459780Z DEBUG slatedb::garbage_collector: Scheduled garbage collection attempt for Compacted SSTs.
2025-06-08T20:19:06.463024Z DEBUG slatedb::garbage_collector: GC has collected 0 Manifests, 0 WAL SSTs and 0 Compacted SSTs.

thread 'tokio-runtime-worker' panicked at src/api/handlers.rs:477:24:
called `Result::unwrap()` on an `Err` value: InvalidFormat("SlateDB error during batch processing")
stack backtrace:
   0:     0x5556a33319d2 - std::backtrace_rs::backtrace::libunwind::trace::hc20b48b31ee52608
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
   1:     0x5556a33319d2 - std::backtrace_rs::backtrace::trace_unsynchronized::h5d207cd20f193d88
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
   2:     0x5556a33319d2 - std::sys::backtrace::_print_fmt::h46402c32145e026c
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:66:9
   3:     0x5556a33319d2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hc04c8f544ab24d66
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:39:26
   4:     0x5556a335e813 - core::fmt::rt::Argument::fmt::h2bcd587ea50ac9a4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/fmt/rt.rs:184:76
   5:     0x5556a335e813 - core::fmt::write::hfe57b7174b7d8eab
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/fmt/mod.rs:1481:21
   6:     0x5556a332dc73 - std::io::default_write_fmt::h20c7e2d50577e95e
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/io/mod.rs:639:11
   7:     0x5556a332dc73 - std::io::Write::write_fmt::h154385efa8565236
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/io/mod.rs:1914:13
   8:     0x5556a3331822 - std::sys::backtrace::BacktraceLock::print::h0c8f24e22f5873a8
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:42:9
   9:     0x5556a33326d2 - std::panicking::default_hook::{{closure}}::hd07d57e6a602c8e4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:300:22
  10:     0x5556a33324d5 - std::panicking::default_hook::h63d12f7d95bd91ed
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:327:9
  11:     0x5556a3333072 - std::panicking::rust_panic_with_hook::h33b18b24045abff4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:833:13
  12:     0x5556a3332e1a - std::panicking::begin_panic_handler::{{closure}}::hf8313cc2fd0126bc
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:706:13
  13:     0x5556a3331ed9 - std::sys::backtrace::__rust_end_short_backtrace::h57fe07c8aea5c98a
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:168:18
  14:     0x5556a3332aad - __rustc[95feac21a9532783]::rust_begin_unwind
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:697:5
  15:     0x5556a2a90170 - core::panicking::panic_fmt::hd54fb667be51beea
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panicking.rs:75:14
  16:     0x5556a2a90596 - core::result::unwrap_failed::he8e27e02739cd3d2
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1704:5
  17:     0x5556a2c6c9ed - core::result::Result<T,E>::unwrap::he7c2f1b26fe28005
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/result.rs:1109:23
  18:     0x5556a2c6c9ed - compactlog::api::handlers::get_entries::{{closure}}::h6d98f7613e594e3c
                               at project/src/api/handlers.rs:477:9
  19:     0x5556a2c6c9ed - <F as axum::handler::Handler<(M,T1,T2),S>>::call::{{closure}}::hc71f266487642d97
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/handler/mod.rs:241:53
  20:     0x5556a2b5dbd1 - <core::pin::Pin<P> as core::future::future::Future>::poll::h37aab4f28224015e
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
  21:     0x5556a2b5dbd1 - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::h7e164de95090df14
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:37
  22:     0x5556a2b5dd7e - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::h0c03d67dc2064131
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
  23:     0x5556a2b5dd7e - <axum::handler::future::IntoServiceFuture<F> as core::future::future::Future>::poll::habf5f31815c63f2c
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/macros.rs:42:17
  24:     0x5556a2b5dd7e - <F as futures_core::future::TryFuture>::try_poll::h9aa6827982a015d6
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-core-0.3.31/src/future.rs:92:9
  25:     0x5556a2b5dd7e - <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll::h37af497a50fc4662
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/try_future/into_future.rs:34:9
  26:     0x5556a2b5dd7e - <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll::hc47b70b1ac90d241
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/future/future/map.rs:55:37
  27:     0x5556a2c77a69 - <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll::hd82613c5f7434699
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
  28:     0x5556a2c77a69 - <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll::h689d03630ec6f7ef
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/futures-util-0.3.31/src/lib.rs:86:13
  29:     0x5556a2c77a69 - <tower::util::map_response::MapResponseFuture<F,N> as core::future::future::Future>::poll::h284c177bfac51f1e
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-0.5.2/src/macros.rs:38:17
  30:     0x5556a2c4d7de - <core::pin::Pin<P> as core::future::future::Future>::poll::h89938290833e0ae9
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/future/future.rs:124:9
  31:     0x5556a2c4d7de - <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h853d89f71838f077
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tower-0.5.2/src/util/oneshot.rs:97:38
  32:     0x5556a2b5d464 - <axum::routing::route::RouteFuture<E> as core::future::future::Future>::poll::h091c35c4edb6a6f8
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/routing/route.rs:182:61
  33:     0x5556a2b5d464 - <hyper_util::service::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll::h2b5aa78ffa287a17
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-util-0.1.13/src/service/oneshot.rs:55:38
  34:     0x5556a2c9344f - <hyper_util::service::glue::TowerToHyperServiceFuture<S,R> as core::future::future::Future>::poll::h96826e6ddfc20c27
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-util-0.1.13/src/service/glue.rs:70:9
  35:     0x5556a2c9344f - <hyper::proto::h1::dispatch::Server<S,hyper::body::incoming::Incoming> as hyper::proto::h1::dispatch::Dispatch>::poll_msg::hc7ddc9d55f23e920
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:538:35
  36:     0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_write::head51ebca3732f4b
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:336:43
  37:     0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_loop::h65f9d22b74946fc9
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:173:21
  38:     0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_inner::h37b0785ce677d5c4
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:149:16
  39:     0x5556a2c9344f - hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T>::poll_catch::h7b1be398515b205c
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:128:28
  40:     0x5556a2b5fcf4 - <hyper::proto::h1::dispatch::Dispatcher<D,Bs,I,T> as core::future::future::Future>::poll::hc7ae0e3611d81f6d
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/proto/h1/dispatch.rs:471:9
  41:     0x5556a2b5fcf4 - <hyper::server::conn::http1::UpgradeableConnection<I,S> as core::future::future::Future>::poll::hfbee6c45f04cc158
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-1.6.0/src/server/conn/http1.rs:530:51
  42:     0x5556a2b0f719 - <hyper_util::server::conn::auto::UpgradeableConnection<I,S,E> as core::future::future::Future>::poll::h07938d6713295559
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/hyper-util-0.1.13/src/server/conn/auto/mod.rs:645:28
  43:     0x5556a2bf4fee - <axum::serve::Serve<M,S> as core::future::into_future::IntoFuture>::into_future::{{closure}}::{{closure}}::h93f4ee28963bf63e
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/axum-0.7.9/src/serve.rs:257:26
  44:     0x5556a2bf2217 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h861cb07729a876aa
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:331:17
  45:     0x5556a2bf2217 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h81c98e46e790608c
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/loom/std/unsafe_cell.rs:16:9
  46:     0x5556a2bf2217 - tokio::runtime::task::core::Core<T,S>::poll::h38841df5f616cca9
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:320:30
  47:     0x5556a2c4901f - tokio::runtime::task::harness::poll_future::{{closure}}::h660f5e872383dcc8
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:532:19
  48:     0x5556a2c4901f - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h317aacec921589f8
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
  49:     0x5556a2c4901f - std::panicking::try::do_call::hc27b14cf34a6773e
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
  50:     0x5556a2c4901f - std::panicking::try::h8130c75a669905b3
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
  51:     0x5556a2c4901f - std::panic::catch_unwind::h93019168ce90aebc
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
  52:     0x5556a2c4901f - tokio::runtime::task::harness::poll_future::h5ead0b51ae04c6f9
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:520:18
  53:     0x5556a2c4901f - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h30f13a22c00db7dd
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:209:27
  54:     0x5556a2c4901f - tokio::runtime::task::harness::Harness<T,S>::poll::h6473ac62d3b781bc
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:154:15
  55:     0x5556a2c4901f - tokio::runtime::task::raw::poll::h4b2fb2e6df391161
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:271:5
  56:     0x5556a31f71f3 - tokio::runtime::task::raw::RawTask::poll::h80a2b40cbfc6d628
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:201:18
  57:     0x5556a31f71f3 - tokio::runtime::task::LocalNotified<S>::run::h0265f707d73c64d9
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/mod.rs:463:9
  58:     0x5556a31f71f3 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h402655e5ffaa975e
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:600:18
  59:     0x5556a31f71f3 - tokio::task::coop::with_budget::h64d1d3518588bab2
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:167:5
  60:     0x5556a31f71f3 - tokio::task::coop::budget::h57c0ef804492b8ae
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/coop/mod.rs:133:5
  61:     0x5556a31f71f3 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h97d0560d2d3bffeb
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:594:9
  62:     0x5556a31f66b0 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h1cd264141ff5c42a
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:542:24
  63:     0x5556a31f0d18 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h9c629e6550bf7295
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:507:21
  64:     0x5556a31f0d18 - tokio::runtime::context::scoped::Scoped<T>::set::hc73416f9295f3423
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/scoped.rs:40:9
  65:     0x5556a3200c68 - tokio::runtime::context::set_scheduler::{{closure}}::hf3cd1df9d97c5171
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:26
  66:     0x5556a3200c68 - std::thread::local::LocalKey<T>::try_with::h787240947063199e
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:311:12
  67:     0x5556a3200c68 - std::thread::local::LocalKey<T>::with::h6c2f0afda759747b
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/local.rs:275:15
  68:     0x5556a3200c68 - tokio::runtime::context::set_scheduler::h726f0c4df2c776eb
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context.rs:176:9
  69:     0x5556a3200c68 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h044afcd2983ac2e9
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:502:9
  70:     0x5556a3200c68 - tokio::runtime::context::runtime::enter_runtime::h671b7106621a36c4
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/context/runtime.rs:65:16
  71:     0x5556a31f5f2d - tokio::runtime::scheduler::multi_thread::worker::run::hf6aeb371ff986937
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:494:5
  72:     0x5556a31ecf59 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h400ccdd1521a3aff
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/multi_thread/worker.rs:460:45
  73:     0x5556a31ecf59 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::h081956de72a1e89e
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/task.rs:42:21
  74:     0x5556a31ecf59 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h47ef6b64ded94dd7
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:331:17
  75:     0x5556a31ecf59 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hb960060d1fad196f
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/loom/std/unsafe_cell.rs:16:9
  76:     0x5556a31ecf59 - tokio::runtime::task::core::Core<T,S>::poll::h94131ad7d3e48ca3
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/core.rs:320:30
  77:     0x5556a31e77b1 - tokio::runtime::task::harness::poll_future::{{closure}}::hc08335a4185f9c18
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:532:19
  78:     0x5556a31e77b1 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h18a1c4cba872391d
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
  79:     0x5556a31e77b1 - std::panicking::try::do_call::h02a74b2f1b882343
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
  80:     0x5556a31e77b1 - std::panicking::try::h261aa36c47cfaed5
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
  81:     0x5556a31e77b1 - std::panic::catch_unwind::heaea8a63fdc403f4
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
  82:     0x5556a31e77b1 - tokio::runtime::task::harness::poll_future::hc4d0e2d2185685a5
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:520:18
  83:     0x5556a31e77b1 - tokio::runtime::task::harness::Harness<T,S>::poll_inner::hcfb3ecb006131488
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:209:27
  84:     0x5556a31e77b1 - tokio::runtime::task::harness::Harness<T,S>::poll::h43dcca0c1d5fe707
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/harness.rs:154:15
  85:     0x5556a320b657 - tokio::runtime::task::raw::RawTask::poll::h80a2b40cbfc6d628
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/raw.rs:201:18
  86:     0x5556a320b657 - tokio::runtime::task::UnownedTask<S>::run::h9ed99ee43f82ff54
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/task/mod.rs:500:9
  87:     0x5556a320b657 - tokio::runtime::blocking::pool::Task::run::he5db632505af0aa5
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/pool.rs:161:9
  88:     0x5556a320b657 - tokio::runtime::blocking::pool::Inner::run::h8ae30c679d447f9b
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/pool.rs:511:17
  89:     0x5556a31f7c40 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h4909d75f996fb58b
                               at /.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/pool.rs:469:13
  90:     0x5556a31f7c40 - std::sys::backtrace::__rust_begin_short_backtrace::h1a39e32b38e38362
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/backtrace.rs:152:18
  91:     0x5556a31f8688 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h4a9daaa72133bd41
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/mod.rs:559:17
  92:     0x5556a31f8688 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h9a9f5fb0a4c36a36
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/panic/unwind_safe.rs:272:9
  93:     0x5556a31f8688 - std::panicking::try::do_call::h2e908e64532c5902
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:589:40
  94:     0x5556a31f8688 - std::panicking::try::he17cf4bdba03423d
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panicking.rs:552:19
  95:     0x5556a31f8688 - std::panic::catch_unwind::h813ba417dc2c2072
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/panic.rs:359:14
  96:     0x5556a31f8688 - std::thread::Builder::spawn_unchecked_::{{closure}}::h39d1083231ffe237
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/thread/mod.rs:557:30
  97:     0x5556a31f8688 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h5642d3d0eb327fd1
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/core/src/ops/function.rs:250:5
  98:     0x5556a333507b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc031c115ad99256b
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1966:9
  99:     0x5556a333507b - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha14f846e5cecd1a7
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/alloc/src/boxed.rs:1966:9
 100:     0x5556a333507b - std::sys::pal::unix::thread::Thread::new::thread_start::h2717d4f9aca577e6
                               at /rustc/17067e9ac6d7ecb70e50f92c1944e545188d2359/library/std/src/sys/pal/unix/thread.rs:109:17
 101:     0x7f7b238c21f5 - <unknown>
 102:     0x7f7b2394289c - <unknown>
 103:                0x0 - <unknown>```

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 20 (14 by maintainers)

Most upvoted comments

Wouldn’t an easier implementation be that the compactor would communicate with the GC through a shared hashset that contains IDs that are part of a current compaction?

We are planning to support compactors, garbage collectors, readers, and the writer each running on a separate machine. Unfortunately, that prevents us from this type of coordination. This is why Rohan was suggesting modifying the manifest.

@rodesai Thinking a bit more about implementation. Regarding the manifest protocol, what do you think about using UUID v7 (time sortable) UUIDs for L0+ and then maintaining a compacted low watermark? The compactor can update the low watermark as it compacts, and the GC isn’t allowed to delete anything > the low watermark UUID (even if it passes min_age).

@criccomini I posted a repro repo just above (just make sure you are on the corruption branch), that should just “work” to produce the same thing after few mins

That shouldn’t be the case. The GC as its implemented looks at all manifests referenced by a checkpoint. list_active_l0_and_compacted_ssts calls manifest_store.read_active_manifests and builds a list of ssts from all the manifests that returens. read_active_manifests should look through all the checkpoints.

There’s a test for this case here: https://github.com/slatedb/slatedb/blob/main/slatedb/src/garbage_collector.rs#L913