risingwave: creation of materialise view stuck with "table is in creating procedure" error

Describe the bug

creation of materialised view got stuck. while retrying the same query I am getting “internal error: table is in creating procedure” error.

You can use this tenant for debugging: ╰─× psql “postgres://dev:dev@rls-apse1-eks-a.risingwave-cloud.xyz:4566/dev?options=–tenant%3Drwc-3-sushant-test”

Logs:

╭─sushantgupta@Sushants-MacBook-Pro in ~/sparrow on ☁️  (ap-southeast-1)
╰─> kubectl logs risingwave-meta-f44ccb9-b9j5m
launching meta node
2022-12-01T05:19:53.834743Z  INFO risingwave_rt: setting tokio worker threads to 8
2022-12-01T05:19:53.835565Z  INFO risingwave_meta: Starting meta node with config MetaNodeConfig { streaming: StreamingConfig { barrier_interval_ms: 250, in_flight_barrier_nums: 40, checkpoint_frequency: 10, minimal_scheduling: false, worker_node_parallelism: 8, actor_runtime_worker_threads_num: None, total_memory_available_bytes: 66716327936, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_enable_managed_cache: true, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } } }
2022-12-01T05:19:53.835591Z  INFO risingwave_meta: Starting meta node with options periodic_compaction_interval_sec: 60, enable_compaction_deterministic: false
2022-12-01T05:19:53.835595Z  INFO risingwave_meta: Meta server listening at 0.0.0.0:5690
2022-12-01T05:19:53.838528Z  INFO risingwave_meta::barrier::schedule: Starting barrier scheduler with: checkpoint_frequency=10
2022-12-01T05:19:53.838555Z  INFO risingwave_meta::barrier: Starting barrier manager with: interval=250ms, enable_recovery=true, in_flight_barrier_nums=40
2022-12-01T05:19:53.838709Z  INFO risingwave_common_service::metrics_manager: Prometheus listener for Prometheus is set up on http://0.0.0.0:1250
2022-12-01T05:19:53.838960Z  INFO risingwave_meta::hummock::compaction_scheduler: Start compaction scheduler.
2022-12-01T05:19:53.839097Z  WARN risingwave_common::util::epoch: New generate epoch is too close to the previous one.
2022-12-01T05:20:07.995579Z  INFO risingwave_meta::hummock::compactor_manager: Added compactor session 3
2022-12-01T05:24:20.062309Z  INFO risingwave_meta::stream::source_manager: spawning new watcher for source 1001
2022-12-01T05:24:20.163942Z  INFO risingwave_meta::stream::source_manager: spawning new watcher for source 1004
2022-12-01T05:24:30.396965Z  INFO risingwave_meta::stream::source_manager: spawning new watcher for source 1007

╭─sushantgupta@Sushants-MacBook-Pro in ~/sparrow on ☁️  (ap-southeast-1)
╰─> kubectl logs risingwave-compactor-79dd4858fc-c7zwx
launching compactor node
2022-12-01T05:19:53.806022Z  INFO risingwave_rt: setting tokio worker threads to 8
2022-12-01T05:19:53.807290Z  INFO risingwave_compactor: meta address: http://risingwave-meta:5690
2022-12-01T05:19:53.807309Z  INFO risingwave_compactor: Server Listening at 10.0.5.240:6660
2022-12-01T05:19:53.807313Z  WARN risingwave_compactor: Client address is not specified, defaulting to host address
2022-12-01T05:19:53.807346Z  INFO risingwave_compactor: Client address is 10.0.5.240:6660
2022-12-01T05:19:53.807443Z  INFO risingwave_compactor::server: Starting compactor with config CompactorConfig { server: ServerConfig { heartbeat_interval_ms: 1000, max_heartbeat_interval_secs: 600, connection_pool_size: 16 }, storage: StorageConfig { sstable_size_mb: 256, block_size_kb: 1024, bloom_false_positive: 0.01, share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: 1024, data_directory: "rls-apse1-eks-a-rwc-3-sushant-test", write_conflict_detection_enabled: false, block_cache_capacity_mb: 256, meta_cache_capacity_mb: 64, disable_remote_compactor: false, enable_local_spill: true, local_object_store: "tempdisk", share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: 512, sstable_id_remote_fetch_number: 10, file_cache: FileCacheConfig { capacity_mb: 1024, total_buffer_capacity_mb: 128, cache_file_fallocate_unit_mb: 512, cache_meta_fallocate_unit_mb: 16, cache_file_max_write_size_mb: 4 }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, object_store_use_batch_delete: true, enable_state_store_v1: false } } and opts CompactorOpts { host: "10.0.5.240:6660", client_address: None, port: None, state_store: "hummock+s3://rls-apse1-mgmt-bucket", prometheus_listener_addr: "0.0.0.0:1260", metrics_level: 1, meta_address: "http://risingwave-meta:5690", config_path: "/risingwave/config/risingwave.toml", max_concurrent_task_number: 16, compaction_worker_threads_number: None }
2022-12-01T05:19:54.831673Z  WARN risingwave_rpc_client::meta_client: Failed to connect to meta server http://risingwave-meta:5690, wait for online: transport error
2022-12-01T05:19:55.951657Z  WARN risingwave_rpc_client::meta_client: Failed to connect to meta server http://risingwave-meta:5690, wait for online: transport error
2022-12-01T05:19:58.831550Z  WARN risingwave_rpc_client::meta_client: Failed to connect to meta server http://risingwave-meta:5690, wait for online: transport error
2022-12-01T05:20:02.959596Z  WARN risingwave_rpc_client::meta_client: Failed to connect to meta server http://risingwave-meta:5690, wait for online: transport error
2022-12-01T05:20:07.912155Z  INFO risingwave_compactor::server: Assigned compactor id 3
2022-12-01T05:20:07.990744Z  INFO risingwave_object_store::object::s3: S3 bucket rls-apse1-mgmt-bucket has already configured AbortIncompleteMultipartUpload: [LifecycleRule { expiration: None, id: Some("abort-incomplete-multipart-upload"), prefix: None, filter: Some(Prefix("")), status: Some(Enabled), transitions: None, noncurrent_version_transitions: None, noncurrent_version_expiration: None, abort_incomplete_multipart_upload: Some(AbortIncompleteMultipartUpload { days_after_initiation: 1 }) }]
2022-12-01T05:20:07.993921Z  INFO risingwave_common_service::metrics_manager: Prometheus listener for Prometheus is set up on http://0.0.0.0:1260

╭─sushantgupta@Sushants-MacBook-Pro in ~/sparrow on ☁️  (ap-southeast-1)
╰─> kubectl logs risingwave-compute-0
launching compute node
2022-12-01T05:19:53.826776Z  INFO risingwave_rt: setting tokio worker threads to 8
2022-12-01T05:19:53.827676Z  INFO risingwave_compute: Compute node options: ComputeNodeOpts { host: "10.0.10.88:5688", client_address: Some("risingwave-compute-0.risingwave-compute:5688"), state_store: "hummock+s3://rls-apse1-mgmt-bucket", prometheus_listener_addr: "0.0.0.0:1222", metrics_level: 1, meta_address: "http://risingwave-meta:5690", config_path: "/risingwave/config/risingwave.toml", enable_jaeger_tracing: false, async_stack_trace: On, file_cache_dir: "", connector_source_endpoint: "127.0.0.1:60061" }
2022-12-01T05:19:53.827711Z  INFO risingwave_compute: Server Listening at 10.0.10.88:5688
2022-12-01T05:19:53.827760Z  INFO risingwave_compute: Client address is risingwave-compute-0.risingwave-compute:5688
2022-12-01T05:19:53.828083Z  INFO risingwave_compute::server: Starting compute node with config ComputeNodeConfig { server: ServerConfig { heartbeat_interval_ms: 1000, max_heartbeat_interval_secs: 600, connection_pool_size: 16 }, batch: BatchConfig { worker_threads_num: None, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_enable_managed_cache: true, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, streaming: StreamingConfig { barrier_interval_ms: 1000, in_flight_barrier_nums: 40, checkpoint_frequency: 10, minimal_scheduling: false, worker_node_parallelism: 8, actor_runtime_worker_threads_num: None, total_memory_available_bytes: 66716327936, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_enable_managed_cache: true, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, storage: StorageConfig { sstable_size_mb: 256, block_size_kb: 1024, bloom_false_positive: 0.01, share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: 1024, data_directory: "rls-apse1-eks-a-rwc-3-sushant-test", write_conflict_detection_enabled: false, block_cache_capacity_mb: 256, meta_cache_capacity_mb: 64, disable_remote_compactor: false, enable_local_spill: true, local_object_store: "tempdisk", share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: 512, sstable_id_remote_fetch_number: 10, file_cache: FileCacheConfig { capacity_mb: 1024, total_buffer_capacity_mb: 128, cache_file_fallocate_unit_mb: 512, cache_meta_fallocate_unit_mb: 16, cache_file_max_write_size_mb: 4 }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, object_store_use_batch_delete: true, enable_state_store_v1: false } } with debug assertions off
2022-12-01T05:19:54.838432Z  WARN risingwave_rpc_client::meta_client: Failed to connect to meta server http://risingwave-meta:5690, wait for online: transport error
2022-12-01T05:19:55.926409Z  WARN risingwave_rpc_client::meta_client: Failed to connect to meta server http://risingwave-meta:5690, wait for online: transport error
2022-12-01T05:20:00.950432Z  WARN risingwave_rpc_client::meta_client: Failed to connect to meta server http://risingwave-meta:5690, wait for online: transport error
2022-12-01T05:20:04.055834Z  INFO risingwave_compute::server: Assigned worker node id 1
2022-12-01T05:20:04.154268Z  INFO risingwave_object_store::object::s3: S3 bucket rls-apse1-mgmt-bucket has already configured AbortIncompleteMultipartUpload: [LifecycleRule { expiration: None, id: Some("abort-incomplete-multipart-upload"), prefix: None, filter: Some(Prefix("")), status: Some(Enabled), transitions: None, noncurrent_version_transitions: None, noncurrent_version_expiration: None, abort_incomplete_multipart_upload: Some(AbortIncompleteMultipartUpload { days_after_initiation: 1 }) }]
2022-12-01T05:20:04.155853Z  INFO risingwave_tracing: tracing service started with slow_request_threshold_ms=100
2022-12-01T05:20:04.160116Z  INFO risingwave_common_service::metrics_manager: Prometheus listener for Prometheus is set up on http://0.0.0.0:1222
2022-12-01T05:24:20.083899Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=32 state=None
2022-12-01T05:24:20.083920Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=28 state=None
2022-12-01T05:24:20.083975Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=26 state=Some([Nexmark(NexmarkSplit { split_index: 1, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083976Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=27 state=Some([Nexmark(NexmarkSplit { split_index: 7, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.084003Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=29 state=None
2022-12-01T05:24:20.084054Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=25 state=None
2022-12-01T05:24:20.084096Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=31 state=None
2022-12-01T05:24:20.084078Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=30 state=Some([Nexmark(NexmarkSplit { split_index: 11, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.099717Z  WARN risingwave_storage::hummock::event_handler::uploader: epoch 3449604921032705 to seal has no data
2022-12-01T05:24:20.215203Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=73 state=Some([Nexmark(NexmarkSplit { split_index: 2, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.215431Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=74 state=None
2022-12-01T05:24:20.215572Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=75 state=Some([Nexmark(NexmarkSplit { split_index: 7, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.215730Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=76 state=Some([Nexmark(NexmarkSplit { split_index: 10, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.215797Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=77 state=Some([Nexmark(NexmarkSplit { split_index: 0, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.215889Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=78 state=None
2022-12-01T05:24:20.215961Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=79 state=Some([Nexmark(NexmarkSplit { split_index: 5, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.216008Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=80 state=None

╭─sushantgupta@Sushants-MacBook-Pro in ~/sparrow on ☁️  (ap-southeast-1)
╰─> kubectl logs risingwave-compute-1
launching compute node
2022-12-01T05:22:02.257411Z  INFO risingwave_rt: setting tokio worker threads to 8
2022-12-01T05:22:02.258281Z  INFO risingwave_compute: Compute node options: ComputeNodeOpts { host: "10.0.5.172:5688", client_address: Some("risingwave-compute-1.risingwave-compute:5688"), state_store: "hummock+s3://rls-apse1-mgmt-bucket", prometheus_listener_addr: "0.0.0.0:1222", metrics_level: 1, meta_address: "http://risingwave-meta:5690", config_path: "/risingwave/config/risingwave.toml", enable_jaeger_tracing: false, async_stack_trace: On, file_cache_dir: "", connector_rpc_endpoint: None }
2022-12-01T05:22:02.258315Z  INFO risingwave_compute: Server Listening at 10.0.5.172:5688
2022-12-01T05:22:02.258368Z  INFO risingwave_compute: Client address is risingwave-compute-1.risingwave-compute:5688
2022-12-01T05:22:02.258749Z  INFO risingwave_compute::server: Starting compute node with config ComputeNodeConfig { server: ServerConfig { heartbeat_interval_ms: 1000, max_heartbeat_interval_secs: 600, connection_pool_size: 16 }, batch: BatchConfig { worker_threads_num: None, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_enable_managed_cache: true, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, streaming: StreamingConfig { barrier_interval_ms: 1000, in_flight_barrier_nums: 40, checkpoint_frequency: 10, minimal_scheduling: false, worker_node_parallelism: 8, actor_runtime_worker_threads_num: None, total_memory_available_bytes: 32597401600, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_enable_managed_cache: true, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, storage: StorageConfig { sstable_size_mb: 256, block_size_kb: 1024, bloom_false_positive: 0.01, share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: 1024, data_directory: "rls-apse1-eks-a-rwc-3-sushant-test", write_conflict_detection_enabled: false, block_cache_capacity_mb: 256, meta_cache_capacity_mb: 64, disable_remote_compactor: false, enable_local_spill: true, local_object_store: "tempdisk", share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: 512, sstable_id_remote_fetch_number: 10, file_cache: FileCacheConfig { capacity_mb: 1024, total_buffer_capacity_mb: 128, cache_file_fallocate_unit_mb: 512, cache_meta_fallocate_unit_mb: 16, cache_file_max_write_size_mb: 4 }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, object_store_use_batch_delete: true, enable_state_store_v1: false } } with debug assertions off
2022-12-01T05:22:02.262339Z  INFO risingwave_compute::server: Assigned worker node id 5
2022-12-01T05:22:02.352678Z  INFO risingwave_object_store::object::s3: S3 bucket rls-apse1-mgmt-bucket has already configured AbortIncompleteMultipartUpload: [LifecycleRule { expiration: None, id: Some("abort-incomplete-multipart-upload"), prefix: None, filter: Some(Prefix("")), status: Some(Enabled), transitions: None, noncurrent_version_transitions: None, noncurrent_version_expiration: None, abort_incomplete_multipart_upload: Some(AbortIncompleteMultipartUpload { days_after_initiation: 1 }) }]
2022-12-01T05:22:02.358944Z  INFO risingwave_common_service::metrics_manager: Prometheus listener for Prometheus is set up on http://0.0.0.0:1222
2022-12-01T05:24:20.083422Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=45 state=None
2022-12-01T05:24:20.083423Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=43 state=None
2022-12-01T05:24:20.083423Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=48 state=None
2022-12-01T05:24:20.083450Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=46 state=Some([Nexmark(NexmarkSplit { split_index: 9, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083488Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=41 state=Some([Nexmark(NexmarkSplit { split_index: 2, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083457Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=47 state=Some([Nexmark(NexmarkSplit { split_index: 8, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083495Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=42 state=Some([Nexmark(NexmarkSplit { split_index: 0, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083575Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=44 state=Some([Nexmark(NexmarkSplit { split_index: 10, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.232817Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=89 state=Some([Nexmark(NexmarkSplit { split_index: 9, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.233031Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=90 state=None
2022-12-01T05:24:20.233160Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=91 state=None
2022-12-01T05:24:20.233285Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=92 state=Some([Nexmark(NexmarkSplit { split_index: 6, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.233430Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=93 state=None
2022-12-01T05:24:20.233553Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=94 state=Some([Nexmark(NexmarkSplit { split_index: 4, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.235117Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=95 state=None
2022-12-01T05:24:20.235288Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=96 state=Some([Nexmark(NexmarkSplit { split_index: 11, split_num: 12, start_offset: None })])

╭─sushantgupta@Sushants-MacBook-Pro in ~/sparrow on ☁️  (ap-southeast-1)
╰─> kubectl logs risingwave-compute-2
launching compute node
2022-12-01T05:22:01.864594Z  INFO risingwave_rt: setting tokio worker threads to 8
2022-12-01T05:22:01.865603Z  INFO risingwave_compute: Compute node options: ComputeNodeOpts { host: "10.0.6.49:5688", client_address: Some("risingwave-compute-2.risingwave-compute:5688"), state_store: "hummock+s3://rls-apse1-mgmt-bucket", prometheus_listener_addr: "0.0.0.0:1222", metrics_level: 1, meta_address: "http://risingwave-meta:5690", config_path: "/risingwave/config/risingwave.toml", enable_jaeger_tracing: false, async_stack_trace: On, file_cache_dir: "", connector_rpc_endpoint: None }
2022-12-01T05:22:01.865634Z  INFO risingwave_compute: Server Listening at 10.0.6.49:5688
2022-12-01T05:22:01.865679Z  INFO risingwave_compute: Client address is risingwave-compute-2.risingwave-compute:5688
2022-12-01T05:22:01.865971Z  INFO risingwave_compute::server: Starting compute node with config ComputeNodeConfig { server: ServerConfig { heartbeat_interval_ms: 1000, max_heartbeat_interval_secs: 600, connection_pool_size: 16 }, batch: BatchConfig { worker_threads_num: None, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_enable_managed_cache: true, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, streaming: StreamingConfig { barrier_interval_ms: 1000, in_flight_barrier_nums: 40, checkpoint_frequency: 10, minimal_scheduling: false, worker_node_parallelism: 8, actor_runtime_worker_threads_num: None, total_memory_available_bytes: 32597401600, developer: DeveloperConfig { batch_output_channel_size: 64, batch_chunk_size: 1024, stream_enable_executor_row_count: false, stream_enable_managed_cache: true, stream_connector_message_buffer_size: 16, unsafe_stream_hash_agg_cache_size: 65536, unsafe_stream_join_cache_size: 65536, unsafe_stream_extreme_cache_size: 1024, stream_chunk_size: 1024 } }, storage: StorageConfig { sstable_size_mb: 256, block_size_kb: 1024, bloom_false_positive: 0.01, share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: 1024, data_directory: "rls-apse1-eks-a-rwc-3-sushant-test", write_conflict_detection_enabled: false, block_cache_capacity_mb: 256, meta_cache_capacity_mb: 64, disable_remote_compactor: false, enable_local_spill: true, local_object_store: "tempdisk", share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: 512, sstable_id_remote_fetch_number: 10, file_cache: FileCacheConfig { capacity_mb: 1024, total_buffer_capacity_mb: 128, cache_file_fallocate_unit_mb: 512, cache_meta_fallocate_unit_mb: 16, cache_file_max_write_size_mb: 4 }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, object_store_use_batch_delete: true, enable_state_store_v1: false } } with debug assertions off
2022-12-01T05:22:01.869361Z  INFO risingwave_compute::server: Assigned worker node id 4
2022-12-01T05:22:01.952412Z  INFO risingwave_object_store::object::s3: S3 bucket rls-apse1-mgmt-bucket has already configured AbortIncompleteMultipartUpload: [LifecycleRule { expiration: None, id: Some("abort-incomplete-multipart-upload"), prefix: None, filter: Some(Prefix("")), status: Some(Enabled), transitions: None, noncurrent_version_transitions: None, noncurrent_version_expiration: None, abort_incomplete_multipart_upload: Some(AbortIncompleteMultipartUpload { days_after_initiation: 1 }) }]
2022-12-01T05:22:01.958275Z  INFO risingwave_common_service::metrics_manager: Prometheus listener for Prometheus is set up on http://0.0.0.0:1222
2022-12-01T05:24:20.083590Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=33 state=None
2022-12-01T05:24:20.083606Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=34 state=None
2022-12-01T05:24:20.083590Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=37 state=None
2022-12-01T05:24:20.083605Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=39 state=None
2022-12-01T05:24:20.083623Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=40 state=Some([Nexmark(NexmarkSplit { split_index: 3, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083615Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=38 state=Some([Nexmark(NexmarkSplit { split_index: 6, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083625Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=36 state=Some([Nexmark(NexmarkSplit { split_index: 4, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.083763Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=35 state=Some([Nexmark(NexmarkSplit { split_index: 5, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.099103Z  WARN risingwave_storage::hummock::event_handler::uploader: epoch 3449604921032705 to seal has no data
2022-12-01T05:24:20.186522Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=81 state=None
2022-12-01T05:24:20.219675Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=82 state=Some([Nexmark(NexmarkSplit { split_index: 8, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.219875Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=83 state=None
2022-12-01T05:24:20.219903Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=84 state=Some([Nexmark(NexmarkSplit { split_index: 3, split_num: 12, start_offset: None })])
2022-12-01T05:24:20.220026Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=85 state=None
2022-12-01T05:24:20.220088Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=86 state=None
2022-12-01T05:24:20.220207Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=87 state=None
2022-12-01T05:24:20.220250Z  INFO risingwave_stream::executor::source::source_executor: start with state actor_id=88 state=Some([Nexmark(NexmarkSplit { split_index: 1, split_num: 12, start_offset: None })])


risingwave-frontend-cb7895fcb-5dkfq

[risingwave-frontend-cb7895fcb-5dkfq] 2022-12-01T05:29:28.340474Z ERROR risingwave_frontend::session: failed to handle sql:
[risingwave-frontend-cb7895fcb-5dkfq] SELECT name FROM  (SELECT pg_catalog.lower(name) AS name FROM pg_catalog.pg_settings   UNION ALL SELECT 'session authorization'   UNION ALL SELECT 'all') ss  WHERE substring(name,1,4)='sour'
[risingwave-frontend-cb7895fcb-5dkfq] LIMIT 1000:
[risingwave-frontend-cb7895fcb-5dkfq] Feature is not yet implemented: pg_catalog.pg_settings is not supported, please use `SHOW` commands for now.
[risingwave-frontend-cb7895fcb-5dkfq] `SHOW TABLES`,
[risingwave-frontend-cb7895fcb-5dkfq] `SHOW MATERIALIZED VIEWS`,
[risingwave-frontend-cb7895fcb-5dkfq] `DESCRIBE <table>`,
[risingwave-frontend-cb7895fcb-5dkfq] `SHOW COLUMNS FROM [table]`
[risingwave-frontend-cb7895fcb-5dkfq]
[risingwave-frontend-cb7895fcb-5dkfq] Tracking issue: https://github.com/risingwavelabs/risingwave/issues/1695
[risingwave-frontend-cb7895fcb-5dkfq] 2022-12-01T05:29:39.701972Z ERROR risingwave_frontend::session: failed to handle sql:
[risingwave-frontend-cb7895fcb-5dkfq] CREATE MATERIALIZED VIEW nexmark_q0
[risingwave-frontend-cb7895fcb-5dkfq] AS
[risingwave-frontend-cb7895fcb-5dkfq] SELECT auction, bidder, price, date_time FROM bid;:
[risingwave-frontend-cb7895fcb-5dkfq] internal error: table is in creating procedure <========

https://buildkite.com/risingwave-test/longevity-test/builds/190

To Reproduce

  1. create a 3 node tenant with user
  2. create a source
CREATE MATERIALIZED SOURCE person (
    "id" BIGINT,
    "name" VARCHAR,
    "email_address" VARCHAR,
    "credit_card" VARCHAR,
    "city" VARCHAR,
    "state" VARCHAR,
    "date_time" TIMESTAMP,
    "extra" VARCHAR,
    PRIMARY KEY (id)
) with (
    connector = 'nexmark',
    nexmark.table.type = 'Person',
    nexmark.split.num = '12',
    nexmark.min.event.gap.in.ns = '0'
) ROW FORMAT JSON;

CREATE MATERIALIZED SOURCE auction (
    "id" BIGINT,
    "item_name" VARCHAR,
    "description" VARCHAR,
    "initial_bid" BIGINT,
    "reserve" BIGINT,
    "date_time" TIMESTAMP,
    "expires" TIMESTAMP,
    "seller" BIGINT,
    "category" BIGINT,
    "extra" VARCHAR,
    PRIMARY KEY (id)
) with (
    connector = 'nexmark',
    nexmark.table.type = 'Auction',
    nexmark.split.num = '12',
    nexmark.min.event.gap.in.ns = '0'
) ROW FORMAT JSON;

CREATE SOURCE bid (
    "auction" BIGINT,
    "bidder" BIGINT,
    "price" BIGINT,
    "channel" VARCHAR,
    "url" VARCHAR,
    "date_time" TIMESTAMP,
    "extra" VARCHAR
) with (
    connector = 'nexmark',
    nexmark.table.type = 'Bid',
    nexmark.split.num = '12',
    nexmark.min.event.gap.in.ns = '0'
) ROW FORMAT JSON;
  1. create an materialise view
CREATE MATERIALIZED VIEW nexmark_q0
AS
SELECT auction, bidder, price, date_time FROM bid;

Creation of materialised got stuck.

Expected behavior

No response

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

Just checked the log here]. Looks like the rwc env uses a different setting: the shared_buffer_capacity_mb is set to 1024M. @wenym1 found out an potential bug that can cause write to be stuck if the write batch size exceeded 0.2 * shared_buffer_capacity_mb. We suspect we have a write batch larger than 0.2 * shared_buffer_capacity_mb here. @wenym1 is working on a fix. In the meantime, can you try setting shared_buffer_capacity_mb to a larger value to see how it behaves?

+3
hzxa21 on Dec 1, 2022

I can successfully reproduce the barrier stuck with shared_buffer_capacity_mb set to 1024. I cannot reproduce the MV creation stuck but I think it is the same cause.

+2
hzxa21 on Dec 1, 2022

The impact seems little under the release build, according to my simple benchmarks. I guess it should be okay for pressure tests like longevity tests. 🤔

+1
BugenZhao on Dec 2, 2022

@fuyufjh seems like it is reproducible in the rwc environment only. Do we do something special in rwc env in terms of setting up the risingwave?

🤔 RWC env still uses in-memory meta store instead of etcd… Could this be a cause? cc. @yezizp2012

🤔 I don’t think so. It only affects failover IIUC. I can give it a try with in-memory meta store locally.

+1
yezizp2012 on Dec 1, 2022

@fuyufjh seems like it is reproducible in the rwc environment only. Do we do something special in rwc env in terms of setting up the risingwave?

🤔 RWC env still uses in-memory meta store instead of etcd… Could this be a cause? cc. @yezizp2012

+1
fuyufjh on Dec 1, 2022
Read more comments on GitHub
← risingwave: bug(debug profile): segfault/EXC_BAD_ACCESS during backtrace capture
risingwave: Failed in get: Hummock error: ObjectStore failed with IO error  →