slatedb: Panic in the compactor when writing to SlateDB from a DataFusion operator

Describe the bug We are attempting to use SlateDb to checkpoint the state of our operators of our Stream Processing system built atop DataFusion.

On a high level here’s how our system works -

  1. We are executing a compute graph that is running on DataFusions tokio based runtime. DataFusion operators implement that ReceiverStream pattern and produce data asynchronously that is routed through the graph by the runtime. (See: https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/stream.rs )

  2. In some of our stateful operators, we want to checkpoint the state

    fn check_point_state_for_epoch(&mut self, epoch_ts: u128) {
        debug!(
            "initiating checkpoint for {} at epoch {}",
            self.channel_tag, epoch_ts
        );
        ...
        let serialized_checkpoint = bincode::serialize(&checkpointed_state).unwrap();
        let key = self.channel_tag.as_bytes();
        let _ = block_on(self.state_backend.put(key, &serialized_checkpoint));
        debug!(
            "finished checkpoint for {} at epoch {}",
            self.channel_tag, epoch_ts
        );
    }

  1. The self.state_backend here is an Arc<Db> instance that is shared between the tasks.

Ever so often, we see a panic while checkpoint that ends up killing the runtime.

For purposes of testing we are using the LocalFilesystem

async fn get_slate_db_backend(path_str: &str) -> Db {
    let object_store: Arc<dyn ObjectStore> = Arc::new(LocalFileSystem::new());
    let options = DbOptions::default();
    debug!("initialising slate db at {}", path_str);
    Db::open_with_opts(Path::from(path_str), options, object_store)
        .await
        .unwrap()
}

And here is the backtrace

thread '<unnamed>' panicked at /Users/amey/.cargo/registry/src/index.crates.io-6f17d22bba15001f/slatedb-0.2.0/src/compactor.rs:159:46:
fatal error loading manifest: ObjectStoreError(JoinError { source: JoinError::Cancelled(Id(109)) })
stack backtrace:
   0:        0x1068a0ef8 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h14dcb4794e811c9f
   1:        0x1068b12d4 - core::fmt::write::h49a9cf68bfb4a321
   2:        0x1068a431c - std::io::Write::write_fmt::h00e4f1b82d71ad85
   3:        0x10688cf74 - std::panicking::default_hook::{{closure}}::hda52b8046522f561
   4:        0x10688cbb0 - std::panicking::default_hook::h9566cb73367aef3e
   5:        0x10688d92c - std::panicking::rust_panic_with_hook::hfa41181adbcf532d
   6:        0x1068a12dc - std::panicking::begin_panic_handler::{{closure}}::ha417e66bf9de7acf
   7:        0x1068a1120 - std::sys::backtrace::__rust_end_short_backtrace::hc7b890f013fa9efb
   8:        0x10688d068 - _rust_begin_unwind
   9:        0x1069eb128 - core::panicking::panic_fmt::h29609029db4d41f4
  10:        0x1069eb0f8 - core::result::unwrap_failed::h1a030347a5d7de67
  11:        0x102ad14d8 - core::result::Result<T,E>::expect::he3596e049c212a86
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/core/src/result.rs:1059:23
  12:        0x102bf6608 - slatedb::compactor::CompactorOrchestrator::run::he520b96bb88ca345
                               at /Users/amey/.cargo/registry/src/index.crates.io-6f17d22bba15001f/slatedb-0.2.0/src/compactor.rs:159:25
  13:        0x1024d87d4 - slatedb::compactor::Compactor::new::{{closure}}::{{closure}}::h37fe734dfc21b1c4
                               at /Users/amey/.cargo/registry/src/index.crates.io-6f17d22bba15001f/slatedb-0.2.0/src/compactor.rs:67:13
  14:        0x1024dadc4 - std::sys::backtrace::__rust_begin_short_backtrace::h68fbd1990538055a
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/sys/backtrace.rs:152:18
  15:        0x10249b704 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc04c79247f926931
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/thread/mod.rs:538:17
  16:        0x10249f2d0 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h666de1a3662eea42
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/core/src/panic/unwind_safe.rs:272:9
  17:        0x1024e7498 - std::panicking::try::do_call::hc5aad5f651f779b4
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/panicking.rs:557:40
  18:        0x10249b918 - ___rust_try
  19:        0x10249b3b4 - std::panicking::try::h50bcaf7edf39fa2a
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/panicking.rs:521:19
  20:        0x10249b3b4 - std::panic::catch_unwind::hcf164ee567674fd1
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/panic.rs:350:14
  21:        0x10249b3b4 - std::thread::Builder::spawn_unchecked_::{{closure}}::ha82d38afd7a6518f
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/thread/mod.rs:537:30
  22:        0x1024ab908 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h134233046647c648
                               at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/core/src/ops/function.rs:250:5
  23:        0x10688e484 - std::sys::pal::unix::thread::Thread::new::thread_start::h3d45bacb8043a502
  24:        0x18d622f94 - __pthread_joiner_wake

Additional context Unclear if this is related to slatedb internals or particularly how SlateDB’s threading is interacting with DataFusion’s runtime.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

@criccomini so the “bug” was caused by an issue elsewhere in our code that was intermittently erroring out on a kafka timeout that said, we did see some panics in the DF runtime even after the fix. Our solution was to Slate DB on a separate runtime and that seems to have taken care of all our issues for now.

I have been meaning to add a follow up PR to make initialising SlateDB with your own runtime a bit more convenient.

it is being set to /tmp/checkpoints/ . currently seeing if how i am trying to use blocking calls to slatedb (perhaps running into timeout issues) from DafaFusion’s streams is the issue but at loss as to why this seems to be an issue with using LFS specifically.

fwiw i do see the manifest and the wal files in the checkpoint directory.

Accidentally opened a PR with my patch. But nevertheless in case it is easier to follow - https://github.com/slatedb/slatedb/pull/253/files