risingwave: storage/streaming: Materialize state table sanity check failed for `Update`

thread 'risingwave-streaming-actor' panicked at 'value mismatch when updating row: Some(b"\x01\xe8\x03\0\0\0\0\0\0\x01\x02\0\0\0\0\0\0\0\x01\x90\xa2\xa5U\0\0\0\0\0\0\0\0\x01\x90\xa2\xa5U\0\0\0\0\0\0\0\0") != [1, 232, 3, 0, 0, 0, 0, 0, 0, 1, 3, 0, 0, 0, 0, 0, 0, 0, 1, 144, 162, 165, 85, 0, 0, 0, 0, 0, 0, 0, 0, 1, 144, 162, 165, 85, 0, 0, 0, 0, 0, 0, 0, 0]', /risingwave/src/storage/src/streaming_table/state_table.rs:473:25
stack backtrace:
thread 'risingwave-streaming-actor' panicked at 'value mismatch when updating row: Some(b"\x01\xe8\x03\0\0\0\0\0\0\x01\x04\0\0\0\0\0\0\0\x01\x82\xa2\xa5U\0\0\0\0\0\0\0\0\x01\x82\xa2\xa5U\0\0\0\0\0\0\0\0") != [1, 232, 3, 0, 0, 0, 0, 0, 0, 1, 8, 0, 0, 0, 0, 0, 0, 0, 1, 130, 162, 165, 85, 0, 0, 0, 0, 0, 0, 0, 0, 1, 130, 162, 165, 85, 0, 0, 0, 0, 0, 0, 0, 0]', /risingwave/src/storage/src/streaming_table/state_table.rs:473:25
thread 'risingwave-streaming-actor' panicked at 'value mismatch when updating row: Some(b"\x01\xe8\x03\0\0\0\0\0\0\x01\x02\0\0\0\0\0\0\0\x01z\xa2\xa5U\0\0\0\0\0\0\0\0\x01z\xa2\xa5U\0\0\0\0\0\0\0\0") != [1, 232, 3, 0, 0, 0, 0, 0, 0, 1, 4, 0, 0, 0, 0, 0, 0, 0, 1, 122, 162, 165, 85, 0, 0, 0, 0, 0, 0, 0, 0, 1, 122, 162, 165, 85, 0, 0, 0, 0, 0, 0, 0, 0]', /risingwave/src/storage/src/streaming_table/state_table.rs:473:25
   0: rust_begin_unwind
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:142:14
   2: risingwave_storage::streaming_table::state_table::StateTable<S>::batch_write_rows::{{closure}}
             at ./src/storage/src/streaming_table/state_table.rs:473:25
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
   4: risingwave_storage::streaming_table::state_table::StateTable<S>::commit::{{closure}}
             at ./src/storage/src/streaming_table/state_table.rs:401:52
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
   6: risingwave_stream::executor::mview::materialize::MaterializeExecutor<S>::execute_inner::{{closure}}
             at ./src/stream/src/executor/mview/materialize.rs:155:59

Full logs here.

May related: #4857 #4774

cc @hzxa21

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 17 (11 by maintainers)

Most upvoted comments

#5065 is the potential cause of this issue. PR is merged now. Let’s wait a few days to see whether this issue still happens.

hzxa21 on Sep 5, 2022

Another occurrence on join executor after #5020. Reproduced steps:

./risedev dev ci-3cn-3fe

sqllogictest -h localhost -p 4565 -h localhost -p 4566 -h localhost -p 4567 -d dev './e2e_test/streaming/**/*.slt' -j 16

commit_id: df00ff4e0

thread 'risingwave-streaming-actor' panicked at 'value mismatch when updating row: Some(b"\x01\xa1\x03\0\0\0\0\0\0\x01S\0\0\0\x01\x01\0\0\0\0\0\0\0") != [1, 161, 3, 0, 0, 0, 0, 0, 0, 1, 83, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0]', /home/patrick/projs/risingwave-dev/src/storage/src/table/streaming_table/state_table.rs:449:25
stack backtrace:
   0: rust_begin_unwind
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/panicking.rs:142:14
   2: risingwave_storage::table::streaming_table::state_table::StateTable<S>::batch_write_rows::{{closure}}
             at ./src/storage/src/table/streaming_table/state_table.rs:449:25
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
   4: risingwave_storage::table::streaming_table::state_table::StateTable<S>::commit::{{closure}}
             at ./src/storage/src/table/streaming_table/state_table.rs:378:52
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
   6: risingwave_stream::executor::managed_state::join::JoinHashMap<K,S>::flush::{{closure}}
             at ./src/stream/src/executor/managed_state/join/mod.rs:340:52
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19
   8: risingwave_stream::executor::hash_join::HashJoinExecutor<K,S,_>::flush_data::{{closure}}
             at ./src/stream/src/executor/hash_join.rs:614:31
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/9067d5277d10f0f32a49ec9c125a33828e26a32b/library/core/src/future/mod.rs:91:19

compute-node-5689.log compute-node-5687.log compute-node-5688.log

hzxa21 on Sep 1, 2022

Can also be related to #4442 cc @xxhZs

hzxa21 on Aug 24, 2022

#5065 is the potential cause of this issue. PR is merged now. Let’s wait a few days to see whether this issue still happens.

So the bug still exists

fuyufjh on Sep 9, 2022