risingwave: Use blocking (foreground) DDL for source and sinks

Background

We know that create mv on table/mv must wait until the backfilling completed by default, otherwise, the newly created MV cannot be consistent with its upstream, which breaks RisingWave’s consistency guarantee as well as PG compatibility.

However, regarding of sink and sources, we have to define the behavior. The acceptable options include:

  1. create mv on table/mv blocks until backfilling completed. create mv on source and create sink returns immediately.
  2. create mv on table/mv/source blocks until backfilling completed. create sink returns immediately.
  3. create mv on table/mv/source and create sink all block until backfilling completed. 🥇

According to a poll in Slack channel, option 3 wins.

Another advantage of blocking DDL is that the errors can be explicitly returned to users. On the contrary, a failure during background DDL happens after the object becomes available to users, thus, it can only be printed to logs, which is much more difficult for users.

Initial motivation: https://risingwave-labs.slack.com/archives/C06B3SEQQQ1/p1708499966310609?thread_ts=1708489235.143729&cid=C06B3SEQQQ1

Tasks

We have already report the errors during DDL to frontend sessions. https://github.com/risingwavelabs/risingwave/pull/13108

The remaining task is to adjust the default behavior - Use blocking (foreground) DDL for source and sinks for both create mv on table/mv/source and create sink by default.

Note that this task only changes the default behavior. If users specify set background_ddl = true manually, we should follow it as usual.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 16 (15 by maintainers)

Most upvoted comments

I think https://github.com/risingwavelabs/risingwave/issues/13035 is a prerequisite for this.

To me this is the main benefit:

Another advantage of blocking DDL is that the errors can be explicitly returned to users. On the contrary, a failure during background DDL happens after the object becomes available to users, thus, it can only be printed to logs, which is much more difficult for users.


Is https://github.com/risingwavelabs/risingwave/pull/14172 a prerequisite for this task?

I suppose so. I think only sources with backfill will support it. For source created w/o backfill, we can just return immediately. So for now it will just be supported for cdc backfill.

Another minor benefit for blocking create-table-with-connector is to cleanup sleep in e2e sqllogictest. Not big enough to affect the plan, but if the feature was finally made possible, we could leverage it.

For kafka backfill, in 1.9 it won’t be the default as mentioned by @xxchan offline. So the create mv on source part will not be urgent to release. Only when it is made default, we should have blocking kafka backfill.

Assigning this issue to @xxchan for the blocking create mv on source part. I will work on the blocking sink part still.

So does it mean that we will now introduce a different behavior to let the source error fail the job specifically when it’s being created?

I think if the goal is better UX (know errors earlier, and no need to check logs), maybe yes. Alternatively, maybe we can do more extensive validation when creating source (maybe even attempt to consuming some events?? not always do-able though). But the challenge I mentioned above (error are hidden very deep) still applies.

Do you have any better ideas for this goal?

+1 that “errors can be explicitly returned to users” is the largest benefit.

I think https://github.com/risingwavelabs/risingwave/issues/13035 is a prerequisite for this.

Similarly for sources, we also tolerate errors extensively. e.g.,

https://github.com/risingwavelabs/risingwave/blob/e3f3fb8f89bbb4879855297f65965fdcc4f8eb9d/src/stream/src/executor/source/source_executor.rs#L438-L443

Note that sometimes the errors are tolerated very deep. e.g., I previously tested that when Kafka broker is killed, rebuild_stream_reader_from_error will not even be called. I guess this is because rdkafka performs retrying internally. This will make it very hard to report errors in a timely manner.

For sinks I’m not sure, maybe it can raise some errors.