slatedb: Panic in the compactor when writing to SlateDB from a DataFusion operator
Describe the bug We are attempting to use SlateDb to checkpoint the state of our operators of our Stream Processing system built atop DataFusion.
On a high level here’s how our system works -
-
We are executing a compute graph that is running on DataFusions tokio based runtime. DataFusion operators implement that ReceiverStream pattern and produce data asynchronously that is routed through the graph by the runtime. (See: https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/stream.rs )
-
In some of our stateful operators, we want to checkpoint the state
fn check_point_state_for_epoch(&mut self, epoch_ts: u128) {
debug!(
"initiating checkpoint for {} at epoch {}",
self.channel_tag, epoch_ts
);
...
let serialized_checkpoint = bincode::serialize(&checkpointed_state).unwrap();
let key = self.channel_tag.as_bytes();
let _ = block_on(self.state_backend.put(key, &serialized_checkpoint));
debug!(
"finished checkpoint for {} at epoch {}",
self.channel_tag, epoch_ts
);
}
- The
self.state_backendhere is anArc<Db>instance that is shared between the tasks.
Ever so often, we see a panic while checkpoint that ends up killing the runtime.
For purposes of testing we are using the LocalFilesystem
async fn get_slate_db_backend(path_str: &str) -> Db {
let object_store: Arc<dyn ObjectStore> = Arc::new(LocalFileSystem::new());
let options = DbOptions::default();
debug!("initialising slate db at {}", path_str);
Db::open_with_opts(Path::from(path_str), options, object_store)
.await
.unwrap()
}
And here is the backtrace
thread '<unnamed>' panicked at /Users/amey/.cargo/registry/src/index.crates.io-6f17d22bba15001f/slatedb-0.2.0/src/compactor.rs:159:46:
fatal error loading manifest: ObjectStoreError(JoinError { source: JoinError::Cancelled(Id(109)) })
stack backtrace:
0: 0x1068a0ef8 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h14dcb4794e811c9f
1: 0x1068b12d4 - core::fmt::write::h49a9cf68bfb4a321
2: 0x1068a431c - std::io::Write::write_fmt::h00e4f1b82d71ad85
3: 0x10688cf74 - std::panicking::default_hook::{{closure}}::hda52b8046522f561
4: 0x10688cbb0 - std::panicking::default_hook::h9566cb73367aef3e
5: 0x10688d92c - std::panicking::rust_panic_with_hook::hfa41181adbcf532d
6: 0x1068a12dc - std::panicking::begin_panic_handler::{{closure}}::ha417e66bf9de7acf
7: 0x1068a1120 - std::sys::backtrace::__rust_end_short_backtrace::hc7b890f013fa9efb
8: 0x10688d068 - _rust_begin_unwind
9: 0x1069eb128 - core::panicking::panic_fmt::h29609029db4d41f4
10: 0x1069eb0f8 - core::result::unwrap_failed::h1a030347a5d7de67
11: 0x102ad14d8 - core::result::Result<T,E>::expect::he3596e049c212a86
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/core/src/result.rs:1059:23
12: 0x102bf6608 - slatedb::compactor::CompactorOrchestrator::run::he520b96bb88ca345
at /Users/amey/.cargo/registry/src/index.crates.io-6f17d22bba15001f/slatedb-0.2.0/src/compactor.rs:159:25
13: 0x1024d87d4 - slatedb::compactor::Compactor::new::{{closure}}::{{closure}}::h37fe734dfc21b1c4
at /Users/amey/.cargo/registry/src/index.crates.io-6f17d22bba15001f/slatedb-0.2.0/src/compactor.rs:67:13
14: 0x1024dadc4 - std::sys::backtrace::__rust_begin_short_backtrace::h68fbd1990538055a
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/sys/backtrace.rs:152:18
15: 0x10249b704 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::hc04c79247f926931
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/thread/mod.rs:538:17
16: 0x10249f2d0 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h666de1a3662eea42
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/core/src/panic/unwind_safe.rs:272:9
17: 0x1024e7498 - std::panicking::try::do_call::hc5aad5f651f779b4
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/panicking.rs:557:40
18: 0x10249b918 - ___rust_try
19: 0x10249b3b4 - std::panicking::try::h50bcaf7edf39fa2a
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/panicking.rs:521:19
20: 0x10249b3b4 - std::panic::catch_unwind::hcf164ee567674fd1
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/panic.rs:350:14
21: 0x10249b3b4 - std::thread::Builder::spawn_unchecked_::{{closure}}::ha82d38afd7a6518f
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/std/src/thread/mod.rs:537:30
22: 0x1024ab908 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h134233046647c648
at /private/tmp/rust-20240905-9990-tmdsun/rustc-1.81.0-src/library/core/src/ops/function.rs:250:5
23: 0x10688e484 - std::sys::pal::unix::thread::Thread::new::thread_start::h3d45bacb8043a502
24: 0x18d622f94 - __pthread_joiner_wake
Additional context Unclear if this is related to slatedb internals or particularly how SlateDB’s threading is interacting with DataFusion’s runtime.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 15 (1 by maintainers)
@criccomini so the “bug” was caused by an issue elsewhere in our code that was intermittently erroring out on a kafka timeout that said, we did see some panics in the DF runtime even after the fix. Our solution was to Slate DB on a separate runtime and that seems to have taken care of all our issues for now.
I have been meaning to add a follow up PR to make initialising SlateDB with your own runtime a bit more convenient.
it is being set to
/tmp/checkpoints/. currently seeing if how i am trying to use blocking calls toslatedb(perhaps running into timeout issues) from DafaFusion’s streams is the issue but at loss as to why this seems to be an issue with using LFS specifically.fwiw i do see the manifest and the wal files in the checkpoint directory.
Accidentally opened a PR with my patch. But nevertheless in case it is easier to follow - https://github.com/slatedb/slatedb/pull/253/files