trino: High CPU usage when reading parquet with wide schema
We have a use case when we store big structures to iceberg tables. Recently we have found a case where a structure has 45K+ nested fields totally. Simple select queries fail with Exceeded CPU limit of 48.00h. For example:
select * from events.trms.trm_crud
where (
laid='3f263312-22f0-22eb-868e-ee4cccb49114'
or taid='3f263312-22f0-22eb-868e-ee4cccb49114'
or luid='3f263312-22f0-22eb-868e-ee4cccb49114'
)
and ts >= from_unixtime(1672531200)
and ts < from_unixtime(1718235603)
order by ts;
Select count() with the same filters return 0 records, which means data filtering consume all the CPU.
We ran CPU profiler on one of the workers and it showed that all the CPU is consumed by IcebergPageSourceProvider.createParquetPageSource method.
There 2 loops which take all the CPU time. In the first one we try to find ColumnChunkMetaData by iterating through all the columns:
https://github.com/trinodb/trino/blob/ad97087b3939818e33511712b48b05419f4463cb/lib/trino-parquet/src/main/java/io/trino/parquet/reader/ParquetReader.java#L496-L500
In the second one we iterate over all the columns to find column index: https://github.com/trinodb/trino/blob/ad97087b3939818e33511712b48b05419f4463cb/lib/trino-parquet/src/main/java/io/trino/parquet/ParquetTypeUtils.java#L134-L151
As we do it for all the primitive fields, totally we call equals/equalsIgnoreCase more than 2B times. And it happens at least for each file (there are 1400 files in the table).
We are discussing changing schema of this table to use JSON. But anyway maybe it worth improving for smaller schemas.
Trino version: 444
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (8 by maintainers)
Sure I will check it