risingwave: Use blocking (foreground) DDL for source and sinks
Background
We know that create mv on table/mv must wait until the backfilling completed by default, otherwise, the newly created MV cannot be consistent with its upstream, which breaks RisingWave’s consistency guarantee as well as PG compatibility.
However, regarding of sink and sources, we have to define the behavior. The acceptable options include:
create mv on table/mvblocks until backfilling completed.create mv on sourceandcreate sinkreturns immediately.create mv on table/mv/sourceblocks until backfilling completed.create sinkreturns immediately.create mv on table/mv/sourceandcreate sinkall block until backfilling completed. 🥇
According to a poll in Slack channel, option 3 wins.
Another advantage of blocking DDL is that the errors can be explicitly returned to users. On the contrary, a failure during background DDL happens after the object becomes available to users, thus, it can only be printed to logs, which is much more difficult for users.
Initial motivation: https://risingwave-labs.slack.com/archives/C06B3SEQQQ1/p1708499966310609?thread_ts=1708489235.143729&cid=C06B3SEQQQ1
Tasks
We have already report the errors during DDL to frontend sessions. https://github.com/risingwavelabs/risingwave/pull/13108
The remaining task is to adjust the default behavior - Use blocking (foreground) DDL for source and sinks for both create mv on table/mv/source and create sink by default.
Note that this task only changes the default behavior. If users specify set background_ddl = true manually, we should follow it as usual.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 16 (15 by maintainers)
I think https://github.com/risingwavelabs/risingwave/issues/13035 is a prerequisite for this.
To me this is the main benefit:
I suppose so. I think only sources with backfill will support it. For source created w/o backfill, we can just return immediately. So for now it will just be supported for cdc backfill.
Another minor benefit for blocking create-table-with-connector is to cleanup
sleepin e2e sqllogictest. Not big enough to affect the plan, but if the feature was finally made possible, we could leverage it.For kafka backfill, in 1.9 it won’t be the default as mentioned by @xxchan offline. So the
create mv on sourcepart will not be urgent to release. Only when it is made default, we should have blocking kafka backfill.Assigning this issue to @xxchan for the
blocking create mv on sourcepart. I will work on theblocking sinkpart still.I think if the goal is better UX (know errors earlier, and no need to check logs), maybe yes. Alternatively, maybe we can do more extensive validation when creating source (maybe even attempt to consuming some events?? not always do-able though). But the challenge I mentioned above (error are hidden very deep) still applies.
Do you have any better ideas for this goal?
+1 that “errors can be explicitly returned to users” is the largest benefit.
Similarly for sources, we also tolerate errors extensively. e.g.,
https://github.com/risingwavelabs/risingwave/blob/e3f3fb8f89bbb4879855297f65965fdcc4f8eb9d/src/stream/src/executor/source/source_executor.rs#L438-L443
Note that sometimes the errors are tolerated very deep. e.g., I previously tested that when Kafka broker is killed,
rebuild_stream_reader_from_errorwill not even be called. I guess this is because rdkafka performs retrying internally. This will make it very hard to report errors in a timely manner.For sinks I’m not sure, maybe it can raise some errors.