vortex: issue: vortex file is too large than parquet

After I convert parquet file to vortex file, I find that vortex file is too large than I expected.

2019809413   seq.parquet
9463239652   seq.vortex

I extract one column which type is struct<struct<list<int64>>>, and this issue can be reproduced, but the compressed vortex array size is 157182932, a little bit bigger than parquet.

150732338  single_column_multigroups.parquet
628437016  single_column_multigroups.vortex

If I slice first 1000 line, the file size is very close.

6315041  single_column_1000.parquet
6577728  single_column_1000.vortex

I just follow quick start to do these things.

I have uploaded test file to google drive - https://drive.google.com/file/d/1Fuk7KUO9PHJzbsmOsGJw2UpzztHeCDU4/view?usp=sharing , and its md5 is e8adb6e279c6a3cfbd2845c5370da869.

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Comments: 16 (11 by maintainers)

Most upvoted comments

We had this before https://github.com/vortex-data/vortex/issues/1798 but it got deleted in compute function migration

Yes, we should add a “compact” function to the ArrayOperations trait.

But no need for new layout writer, repartition should call it