risingwave: panic when using Iceberg Sink connector from materialized view

Describe the bug

The sink connector to Iceberg table in hive catalog is throwing error. It is suppose to use the materialized view to sink data into Iceberg table, but risingwave is not able to support the mechanism.

Error message/log

SQL Error [XX000]: ERROR: Panicked when handling the request: called `Result::unwrap()` on an `Err` value: JavaException
This is a bug. We would appreciate a bug report at:
  https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Fbug&template=bug_report.yml

org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [XX000]: ERROR: Panicked when handling the request: called `Result::unwrap()` on an `Err` value: JavaException
This is a bug. We would appreciate a bug report at:
  https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Fbug&template=bug_report.yml
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:133)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeStatement(SQLQueryJob.java:614)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.lambda$2(SQLQueryJob.java:505)
	at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:191)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeSingleQuery(SQLQueryJob.java:524)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.extractData(SQLQueryJob.java:976)
	at org.jkiss.dbeaver.ui.editors.sql.SQLEditor$QueryResultsContainer.readData(SQLEditor.java:4133)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:123)
	at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:191)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:121)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$ResultSetDataPumpJob.run(ResultSetViewer.java:5148)
	at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:114)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: org.postgresql.util.PSQLException: ERROR: Panicked when handling the request: called `Result::unwrap()` on an `Err` value: JavaException
This is a bug. We would appreciate a bug report at:
  https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Fbug&template=bug_report.yml
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:341)
	at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:326)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:302)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:297)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:330)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:131)
	... 12 more

To Reproduce

--------------Create a table that fetch the data fromo Google pubsub as a source create table public.segment_impressions ( “anonymousId” varchar, “context” jsonb, “event” varchar, “integrations” jsonb, “messageId” varchar, “originalTimestamp” varchar, “properties” jsonb, “receivedAt” varchar, “sentAt” varchar, “timestamp” varchar, “type” varchar, “writeKey” varchar ) WITH ( connector = ‘google_pubsub’, pubsub.subscription = ‘risingwave-test’, pubsub.credentials = ‘<google service account description>’ ) FORMAT PLAIN ENCODE JSON;

----------Create a materialized view that transforms the data flowing from the source create materialized view public.segment_impression_event_mv (appointment_id, auction_id, tvcDealerId, timestamp_ist) as select replace((json_data->‘appointment_id’)::varchar,‘"’,‘’) as appointment_id, replace((json_data->‘auction_id’)::varchar,‘"’,‘’) as auction_id, tvcDealerId, timestamp_ist from( select jsonb_array_elements((properties->‘car_info’ #>> ‘{}’)::jsonb) as json_data, replace((properties->‘tvcDealerId’)::varchar,‘"’,‘’) as tvcDealerId , to_timestamp(substring(replace(timestamp,‘T’,’ ‘),1,19),‘YYYY-MM-DD HH24:MI:SS’)::timestamp without time zone + INTERVAL ‘330 MINUTES’ as timestamp_ist from public.segment_impressions where to_timestamp(substring(replace(timestamp,‘T’,’ '),1,19),‘YYYY-MM-DD HH24:MI:SS’)::timestamp without time zone + INTERVAL ‘330 MINUTES’ >= current_timestamp - interval ‘2 DAYS’ ) as tbl;

---------------Sink the transformed data in materialized view to and Iceberg table in Hive catalog CREATE SINK public.segment_impression_data FROM public.segment_impression_event_mv WITH ( connector = ‘iceberg’, type = ‘append-only’, force_append_only = ‘true’, warehouse.path = ‘s3a://hive-iceberg/segment_db/’, s3.endpoint = ‘http://X.X.X.X’, s3.access.key = ‘', s3.secret.key = '************’, s3.region = ‘asia-south1’, – catalog.name = ‘iceberg_hive_qa’, catalog.type = ‘hive’, catalog.uri = ‘thrift://X.X.X.X:9083’, database.name = ‘segmentdb’, table.name = ‘segment_impression_data_rw’ – primary_key=‘seq_id’ );

---------------This is the part where the bug is while loading the MV data into Iceberg sink connector.

Expected behavior

I expected to see the data flowing from materialized view to and Iceberg table in hive catalog, instead it is not able to configure or read metadata of the iceberg table to setup the connection. The I/O operation is failing and even if the command is syntactically correct the sink connector operation is failing with no proper error details.

How did you deploy RisingWave?

Deployed on GKE through operator. My risingwave-operator-system yaml file is: apiVersion: v1 kind: Service metadata: name: risingwave-etcd labels: app: risingwave-etcd spec: ports:

  • port: 2388 name: client
  • port: 2389 name: peer selector: app: risingwave-etcd

apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: risingwave-etcd name: risingwave-etcd spec: replicas: 1 selector: matchLabels: app: risingwave-etcd serviceName: risingwave-etcd volumeClaimTemplates:

  • metadata: name: etcd-data spec: accessModes: [ “ReadWriteOnce” ] resources: requests: storage: 10Gi persistentVolumeClaimRetentionPolicy: whenDeleted: Delete whenScaled: Retain template: metadata: labels: app: risingwave-etcd spec: nodeSelector: cloud.google.com/gke-nodepool: risingwave-pool containers: - name: etcd image: quay.io/coreos/etcd:latest imagePullPolicy: IfNotPresent command: - /usr/local/bin/etcd args: - “–listen-client-urls” - “http://0.0.0.0:2388/” - “–advertise-client-urls” - “http://risingwave-etcd-0:2388/” - “–listen-peer-urls” - “http://0.0.0.0:2389/” - “–initial-advertise-peer-urls” - “http://risingwave-etcd-0:2389/” - “–listen-metrics-urls” - “http://0.0.0.0:2379/” - “–name” - “risingwave-etcd” - “–max-txn-ops” - “999999” - “–max-request-bytes” - “104857600” - “–auto-compaction-mode” - periodic - “–auto-compaction-retention” - 1m - “–snapshot-count” - “10000” - --quota-backend-bytes - “8589934592” - --data-dir - /var/lib/etcd env: - name: ALLOW_NONE_AUTHENTICATION value: “1” ports: - containerPort: 2389 name: peer protocol: TCP - containerPort: 2388 name: client protocol: TCP volumeMounts: - mountPath: /var/lib/etcd name: etcd-data

apiVersion: v1 kind: Secret metadata: name: gcs-credentials stringData: ServiceAccountCredentials: “<creds>”

apiVersion: risingwave.risingwavelabs.com/v1alpha1 kind: RisingWave metadata: name: risingwave-etcd-gcs spec: metaStore: etcd: endpoint: risingwave-etcd:2388 stateStore: gcs: bucket: risingwave-test root: risingwave credentials: secretName: gcs-credentials serviceAccountCredentialsKeyRef: ServiceAccountCredentials image: risingwavelabs/risingwave:v1.7.3 components: meta: nodeGroups: - replicas: 1 name: “” template: spec: nodeSelector: cloud.google.com/gke-nodepool: risingwave-pool volumes: - name: heap emptyDir: sizeLimit: 1Gi volumeMounts: - mountPath: /heap name: heap env: - name: MALLOC_CONF value: prof:true,lg_prof_interval=-1,lg_prof_sample=20,prof_prefix:/heap/ - name: RW_HEAP_PROFILING_DIR value: /heap resources: limits: cpu: 1 memory: 2Gi requests: cpu: 1 memory: 2Gi frontend: nodeGroups: - replicas: 1 name: “” template: spec: nodeSelector: cloud.google.com/gke-nodepool: risingwave-pool resources: limits: cpu: 1 memory: 2Gi requests: cpu: 1 memory: 2Gi compute: nodeGroups: - replicas: 1 name: “” template: spec: nodeSelector: cloud.google.com/gke-nodepool: risingwave-pool volumes: - name: heap emptyDir: sizeLimit: 1Gi volumeMounts: - mountPath: /heap name: heap env: - name: MALLOC_CONF value: prof:true,lg_prof_interval=-1,lg_prof_sample=20,prof_prefix:/heap/ - name: RW_HEAP_PROFILING_DIR value: /heap resources: limits: cpu: 4 memory: 16Gi # Memory limit will be set to RW_TOTAL_MEMORY_BYTES requests: cpu: 4 memory: 16Gi compactor: nodeGroups: - replicas: 1 name: “” template: spec: nodeSelector: cloud.google.com/gke-nodepool: risingwave-pool volumes: - name: heap emptyDir: sizeLimit: 1Gi volumeMounts: - mountPath: /heap name: heap env: - name: MALLOC_CONF value: prof:true,lg_prof_interval=-1,lg_prof_sample=20,prof_prefix:/heap/ - name: RW_HEAP_PROFILING_DIR value: /heap resources: limits: cpu: 2 memory: 4Gi requests: cpu: 2 memory: 4Gi

The version of RisingWave

PostgreSQL 9.5.0-RisingWave-1.7.3 (cfefe78401f38993692e761418e1062ae16799a1)

Additional context

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 22 (11 by maintainers)

Most upvoted comments

Thanks for the reporting @anubhavgupta2404. Could you please use our latest version v1.9.1 to try again?