slatedb: Notify clients when there is a failure

Client puts and deletes don’t have the opportunity to receive SlateDBError::Fenced errors. How do clients detect when they’ve been fenced?

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 17 (5 by maintainers)

Commits related to this issue

Most upvoted comments

Sorry something came up this week. I will try to finish this up tomorrow or day after if I can’t make a PR by then we can have someone else try this

On Sat, Oct 5, 2024, 4:56 AM Chris Riccomini @.***> wrote:

@palash25 https://github.com/palash25 Checking in on this one

— Reply to this email directly, view it on GitHub https://github.com/slatedb/slatedb/issues/111#issuecomment-2394784601, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFDAXHRKPSRDGIFSXO34DALZZ4PZJAVCNFSM6AAAAABMLCCG3WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJUG44DINRQGE . You are receiving this because you were mentioned.Message ID: @.***>

starting work on this

sure, this seems interesting.

@dyedgreen you should also join the SlateDB Discord Server if you haven’t already. It would be great to have you there 🙂

Those two look good as well; happy to get started with #77 (from reading the issue that one seems a bit easier, so might be a better one to get started with? will also comment on that issue to prevent further races 🫣)

There is more discussion about this issue on Discord. The general idea is that we have several background threads: WAL flusher, memtable flusher, and compaction. If errors occur on these threads, they die, but the user is never notified of the failure. They continue put’ing and get’ing and weird behavior can happen.

Per-the discussion, I think the easiest path forward is:

  1. Introduce a error: Option<SlateDbError> in DbState.
  2. Replace all panic!()'s by setting the DbState error.
  3. Catch any errors in the base of the background threads (flush, memtable_flush, and compaction_executor/scheduler) and set the DbState error.
  4. When set, the various user-facing APIs should return that error rather than proceeding with the method calls.