vortex: issue: vortex file is too large than parquet
After I convert parquet file to vortex file, I find that vortex file is too large than I expected.
2019809413 seq.parquet
9463239652 seq.vortex
I extract one column which type is struct<struct<list<int64>>>, and this issue can be reproduced, but the compressed vortex array size is 157182932, a little bit bigger than parquet.
150732338 single_column_multigroups.parquet
628437016 single_column_multigroups.vortex
If I slice first 1000 line, the file size is very close.
6315041 single_column_1000.parquet
6577728 single_column_1000.vortex
I just follow quick start to do these things.
I have uploaded test file to google drive - https://drive.google.com/file/d/1Fuk7KUO9PHJzbsmOsGJw2UpzztHeCDU4/view?usp=sharing , and its md5 is e8adb6e279c6a3cfbd2845c5370da869.
About this issue
- Original URL
- State: open
- Created a year ago
- Comments: 16 (11 by maintainers)
We had this before https://github.com/vortex-data/vortex/issues/1798 but it got deleted in compute function migration
Yes, we should add a “compact” function to the ArrayOperations trait.
But no need for new layout writer, repartition should call it