risingwave: panic when using Iceberg Sink connector from materialized view
Describe the bug
The sink connector to Iceberg table in hive catalog is throwing error. It is suppose to use the materialized view to sink data into Iceberg table, but risingwave is not able to support the mechanism.
Error message/log
SQL Error [XX000]: ERROR: Panicked when handling the request: called `Result::unwrap()` on an `Err` value: JavaException
This is a bug. We would appreciate a bug report at:
https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Fbug&template=bug_report.yml
org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [XX000]: ERROR: Panicked when handling the request: called `Result::unwrap()` on an `Err` value: JavaException
This is a bug. We would appreciate a bug report at:
https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Fbug&template=bug_report.yml
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:133)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeStatement(SQLQueryJob.java:614)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.lambda$2(SQLQueryJob.java:505)
at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:191)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeSingleQuery(SQLQueryJob.java:524)
at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.extractData(SQLQueryJob.java:976)
at org.jkiss.dbeaver.ui.editors.sql.SQLEditor$QueryResultsContainer.readData(SQLEditor.java:4133)
at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:123)
at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:191)
at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:121)
at org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$ResultSetDataPumpJob.run(ResultSetViewer.java:5148)
at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:114)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: org.postgresql.util.PSQLException: ERROR: Panicked when handling the request: called `Result::unwrap()` on an `Err` value: JavaException
This is a bug. We would appreciate a bug report at:
https://github.com/risingwavelabs/risingwave/issues/new?labels=type%2Fbug&template=bug_report.yml
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:371)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:502)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:419)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:341)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:326)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:302)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:297)
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:330)
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:131)
... 12 more
To Reproduce
--------------Create a table that fetch the data fromo Google pubsub as a source create table public.segment_impressions ( “anonymousId” varchar, “context” jsonb, “event” varchar, “integrations” jsonb, “messageId” varchar, “originalTimestamp” varchar, “properties” jsonb, “receivedAt” varchar, “sentAt” varchar, “timestamp” varchar, “type” varchar, “writeKey” varchar ) WITH ( connector = ‘google_pubsub’, pubsub.subscription = ‘risingwave-test’, pubsub.credentials = ‘<google service account description>’ ) FORMAT PLAIN ENCODE JSON;
----------Create a materialized view that transforms the data flowing from the source create materialized view public.segment_impression_event_mv (appointment_id, auction_id, tvcDealerId, timestamp_ist) as select replace((json_data->‘appointment_id’)::varchar,‘"’,‘’) as appointment_id, replace((json_data->‘auction_id’)::varchar,‘"’,‘’) as auction_id, tvcDealerId, timestamp_ist from( select jsonb_array_elements((properties->‘car_info’ #>> ‘{}’)::jsonb) as json_data, replace((properties->‘tvcDealerId’)::varchar,‘"’,‘’) as tvcDealerId , to_timestamp(substring(replace(timestamp,‘T’,’ ‘),1,19),‘YYYY-MM-DD HH24:MI:SS’)::timestamp without time zone + INTERVAL ‘330 MINUTES’ as timestamp_ist from public.segment_impressions where to_timestamp(substring(replace(timestamp,‘T’,’ '),1,19),‘YYYY-MM-DD HH24:MI:SS’)::timestamp without time zone + INTERVAL ‘330 MINUTES’ >= current_timestamp - interval ‘2 DAYS’ ) as tbl;
---------------Sink the transformed data in materialized view to and Iceberg table in Hive catalog CREATE SINK public.segment_impression_data FROM public.segment_impression_event_mv WITH ( connector = ‘iceberg’, type = ‘append-only’, force_append_only = ‘true’, warehouse.path = ‘s3a://hive-iceberg/segment_db/’, s3.endpoint = ‘http://X.X.X.X’, s3.access.key = ‘', s3.secret.key = '************’, s3.region = ‘asia-south1’, – catalog.name = ‘iceberg_hive_qa’, catalog.type = ‘hive’, catalog.uri = ‘thrift://X.X.X.X:9083’, database.name = ‘segmentdb’, table.name = ‘segment_impression_data_rw’ – primary_key=‘seq_id’ );
---------------This is the part where the bug is while loading the MV data into Iceberg sink connector.
Expected behavior
I expected to see the data flowing from materialized view to and Iceberg table in hive catalog, instead it is not able to configure or read metadata of the iceberg table to setup the connection. The I/O operation is failing and even if the command is syntactically correct the sink connector operation is failing with no proper error details.
How did you deploy RisingWave?
Deployed on GKE through operator. My risingwave-operator-system yaml file is: apiVersion: v1 kind: Service metadata: name: risingwave-etcd labels: app: risingwave-etcd spec: ports:
- port: 2388 name: client
- port: 2389 name: peer selector: app: risingwave-etcd
apiVersion: apps/v1 kind: StatefulSet metadata: labels: app: risingwave-etcd name: risingwave-etcd spec: replicas: 1 selector: matchLabels: app: risingwave-etcd serviceName: risingwave-etcd volumeClaimTemplates:
- metadata: name: etcd-data spec: accessModes: [ “ReadWriteOnce” ] resources: requests: storage: 10Gi persistentVolumeClaimRetentionPolicy: whenDeleted: Delete whenScaled: Retain template: metadata: labels: app: risingwave-etcd spec: nodeSelector: cloud.google.com/gke-nodepool: risingwave-pool containers: - name: etcd image: quay.io/coreos/etcd:latest imagePullPolicy: IfNotPresent command: - /usr/local/bin/etcd args: - “–listen-client-urls” - “http://0.0.0.0:2388/” - “–advertise-client-urls” - “http://risingwave-etcd-0:2388/” - “–listen-peer-urls” - “http://0.0.0.0:2389/” - “–initial-advertise-peer-urls” - “http://risingwave-etcd-0:2389/” - “–listen-metrics-urls” - “http://0.0.0.0:2379/” - “–name” - “risingwave-etcd” - “–max-txn-ops” - “999999” - “–max-request-bytes” - “104857600” - “–auto-compaction-mode” - periodic - “–auto-compaction-retention” - 1m - “–snapshot-count” - “10000” - --quota-backend-bytes - “8589934592” - --data-dir - /var/lib/etcd env: - name: ALLOW_NONE_AUTHENTICATION value: “1” ports: - containerPort: 2389 name: peer protocol: TCP - containerPort: 2388 name: client protocol: TCP volumeMounts: - mountPath: /var/lib/etcd name: etcd-data
apiVersion: v1 kind: Secret metadata: name: gcs-credentials stringData: ServiceAccountCredentials: “<creds>”
apiVersion: risingwave.risingwavelabs.com/v1alpha1
kind: RisingWave
metadata:
name: risingwave-etcd-gcs
spec:
metaStore:
etcd:
endpoint: risingwave-etcd:2388
stateStore:
gcs:
bucket: risingwave-test
root: risingwave
credentials:
secretName: gcs-credentials
serviceAccountCredentialsKeyRef: ServiceAccountCredentials
image: risingwavelabs/risingwave:v1.7.3
components:
meta:
nodeGroups:
- replicas: 1
name: “”
template:
spec:
nodeSelector:
cloud.google.com/gke-nodepool: risingwave-pool
volumes:
- name: heap
emptyDir:
sizeLimit: 1Gi
volumeMounts:
- mountPath: /heap
name: heap
env:
- name: MALLOC_CONF
value: prof:true,lg_prof_interval=-1,lg_prof_sample=20,prof_prefix:/heap/
- name: RW_HEAP_PROFILING_DIR
value: /heap
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
frontend:
nodeGroups:
- replicas: 1
name: “”
template:
spec:
nodeSelector:
cloud.google.com/gke-nodepool: risingwave-pool
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
compute:
nodeGroups:
- replicas: 1
name: “”
template:
spec:
nodeSelector:
cloud.google.com/gke-nodepool: risingwave-pool
volumes:
- name: heap
emptyDir:
sizeLimit: 1Gi
volumeMounts:
- mountPath: /heap
name: heap
env:
- name: MALLOC_CONF
value: prof:true,lg_prof_interval=-1,lg_prof_sample=20,prof_prefix:/heap/
- name: RW_HEAP_PROFILING_DIR
value: /heap
resources:
limits:
cpu: 4
memory: 16Gi # Memory limit will be set to RW_TOTAL_MEMORY_BYTES
requests:
cpu: 4
memory: 16Gi
compactor:
nodeGroups:
- replicas: 1
name: “”
template:
spec:
nodeSelector:
cloud.google.com/gke-nodepool: risingwave-pool
volumes:
- name: heap
emptyDir:
sizeLimit: 1Gi
volumeMounts:
- mountPath: /heap
name: heap
env:
- name: MALLOC_CONF
value: prof:true,lg_prof_interval=-1,lg_prof_sample=20,prof_prefix:/heap/
- name: RW_HEAP_PROFILING_DIR
value: /heap
resources:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 2
memory: 4Gi
The version of RisingWave
PostgreSQL 9.5.0-RisingWave-1.7.3 (cfefe78401f38993692e761418e1062ae16799a1)
Additional context
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 22 (11 by maintainers)
Thanks for the reporting @anubhavgupta2404. Could you please use our latest version v1.9.1 to try again?