trino: Problem with ACID table creation

Hi, I tried to create table with ‘transactional’ property set to TRUE with Trino, but I got error message. My environment is:

  • Hive Metastore 3.0 (as standalone)
  • MinIO version: 2021-02-01T22:56:52Z
  • Trino version 352

Query running on Trino:

create table hive3."default".test_acid (id varchar, msg varchar) with (
	format='ORC',
	external_location = 's3n://sbox-jkul/test-acid',
	transactional=true
);

Error message which I get from Trino:

org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [16777216]: Query failed (#20210222_062419_01791_v5qpp): The table must be stored using an ACID compliant format (such as ORC): default.test_acid
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:134)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeStatement(SQLQueryJob.java:481)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.lambda$0(SQLQueryJob.java:420)
	at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:154)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeSingleQuery(SQLQueryJob.java:412)
	at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.extractData(SQLQueryJob.java:771)
	at org.jkiss.dbeaver.ui.editors.sql.SQLEditor$QueryResultsContainer.readData(SQLEditor.java:2859)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:111)
	at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:154)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:109)
	at org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$17.run(ResultSetViewer.java:3434)
	at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:103)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: java.sql.SQLException: Query failed (#20210222_062419_01791_v5qpp): The table must be stored using an ACID compliant format (such as ORC): default.test_acid
	at io.trino.jdbc.AbstractTrinoResultSet.resultsException(AbstractTrinoResultSet.java:1912)
	at io.trino.jdbc.TrinoResultSet.getColumns(TrinoResultSet.java:242)
	at io.trino.jdbc.TrinoResultSet.create(TrinoResultSet.java:53)
	at io.trino.jdbc.TrinoStatement.internalExecute(TrinoStatement.java:249)
	at io.trino.jdbc.TrinoStatement.execute(TrinoStatement.java:227)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:338)
	at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:131)
	... 12 more
Caused by: io.trino.spi.TrinoException: The table must be stored using an ACID compliant format (such as ORC): default.test_acid
	at io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore.createTable(ThriftHiveMetastore.java:1080)
	at io.trino.plugin.hive.metastore.thrift.BridgingHiveMetastore.createTable(BridgingHiveMetastore.java:204)
	at io.trino.plugin.hive.metastore.cache.CachingHiveMetastore.createTable(CachingHiveMetastore.java:508)
	at io.trino.plugin.hive.HiveMetastoreClosure.createTable(HiveMetastoreClosure.java:149)
	at io.trino.plugin.hive.metastore.SemiTransactionalHiveMetastore$CreateTableOperation.run(SemiTransactionalHiveMetastore.java:3050)
	at io.trino.plugin.hive.metastore.SemiTransactionalHiveMetastore$Committer.executeAddTableOperations(SemiTransactionalHiveMetastore.java:1955)
	at io.trino.plugin.hive.metastore.SemiTransactionalHiveMetastore.commitShared(SemiTransactionalHiveMetastore.java:1387)
	at io.trino.plugin.hive.metastore.SemiTransactionalHiveMetastore.commit(SemiTransactionalHiveMetastore.java:1143)
	at io.trino.plugin.hive.HiveMetadata.commit(HiveMetadata.java:2911)
	at io.trino.plugin.hive.HiveConnector.commit(HiveConnector.java:212)
	at io.trino.transaction.InMemoryTransactionManager$TransactionMetadata$ConnectorTransactionMetadata.commit(InMemoryTransactionManager.java:594)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
	at io.trino.$gen.Trino_352____20210212_071926_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: The table must be stored using an ACID compliant format (such as ORC): default.test_acid
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result$create_table_resultStandardScheme.read(ThriftHiveMetastore.java:54068)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result$create_table_resultStandardScheme.read(ThriftHiveMetastore.java:54036)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_result.read(ThriftHiveMetastore.java:53962)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table(ThriftHiveMetastore.java:1531)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table(ThriftHiveMetastore.java:1518)
	at io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient.createTable(ThriftHiveMetastoreClient.java:172)
	at io.trino.plugin.hive.metastore.thrift.FailureAwareThriftMetastoreClient.lambda$createTable$7(FailureAwareThriftMetastoreClient.java:132)
	at io.trino.plugin.hive.metastore.thrift.FailureAwareThriftMetastoreClient.runWithHandle(FailureAwareThriftMetastoreClient.java:467)
	at io.trino.plugin.hive.metastore.thrift.FailureAwareThriftMetastoreClient.createTable(FailureAwareThriftMetastoreClient.java:132)
	at io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore.lambda$createTable$51(ThriftHiveMetastore.java:1068)
	at io.trino.plugin.hive.metastore.thrift.ThriftMetastoreApiStats.lambda$wrap$0(ThriftMetastoreApiStats.java:42)
	at io.trino.plugin.hive.util.RetryDriver.run(RetryDriver.java:130)
	at io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastore.createTable(ThriftHiveMetastore.java:1066)
	... 19 more

Is this bug or I doing something wrong?

Thanks for help.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (7 by maintainers)

Most upvoted comments

I don’t know what’s going on, but here is the information I’ve gathered:

The error message The table must be stored using an ACID compliant format (such as ORC) apparently comes from this chunk of code in Hive metastore method org.apache.hadoop.hive.metastore.TransactionalValidationListener.handleAlterTableTransactionalProp(). The exception is thrown at line 171 in Hive tip master:

    if ("true".equalsIgnoreCase(transactionalValue) && !"true".equalsIgnoreCase(oldTransactionalValue)) {
      if(!isTransactionalPropertiesPresent) {
        normalizeTransactionalPropertyDefault(newTable);
        isTransactionalPropertiesPresent = true;
        transactionalPropertiesValue = DEFAULT_TRANSACTIONAL_PROPERTY;
      }
      // We only need to check conformance if alter table enabled acid.
      // INSERT_ONLY tables don't have to conform to ACID requirement like ORC or bucketing.
      boolean isFullAcid = transactionalPropertiesValue == null
          || !"insert_only".equalsIgnoreCase(transactionalPropertiesValue);
      if (isFullAcid && !conformToAcid(newTable)) {
        throw new MetaException("The table must be stored using an ACID compliant "
            + "format (such as ORC): " + Warehouse.getQualifiedName(newTable));
      }
      ...

conformToAcid() has this definition:

  public static boolean conformToAcid(Table table) throws MetaException {
    StorageDescriptor sd = table.getSd();
    try {
      Class inputFormatClass = sd.getInputFormat() == null ? null :
          Class.forName(sd.getInputFormat());
      Class outputFormatClass = sd.getOutputFormat() == null ? null :
          Class.forName(sd.getOutputFormat());

      if (inputFormatClass == null || outputFormatClass == null ||
          !Class.forName("org.apache.hadoop.hive.ql.io.AcidInputFormat").isAssignableFrom(inputFormatClass) ||
          !Class.forName("org.apache.hadoop.hive.ql.io.AcidOutputFormat").isAssignableFrom(outputFormatClass)) {
        return false;
      }
    } catch (ClassNotFoundException e) {
      LOG.warn("Could not verify InputFormat=" + sd.getInputFormat() + " or OutputFormat=" +
          sd.getOutputFormat() + "  for " + Warehouse.getQualifiedName(table));
      return false;
    }

    return true;
  }

Perhaps the ClassDefNotFound exception is being thrown? Could you check the Hive metastore logs for the “Could not verify InputFormat” warning?

+1
djsstarburst on Apr 23, 2021
Read more comments on GitHub
← trino: node.environment: may not be null
trino: Proper truncation missing in casts to varchar(x) →