arrow: [R][C++] "negative buffer resize" error with arrow and dplyr in R
Hi everyone,
I was working on a large dataset with over 1 billion observations, stored in 3040 parquet files, with 41 variables. I read the data with open_dataset() and then wanted to apply dplyr functions:
individual_positions %>%
group_by(user_id) %>%
summarize(n_positions = n()) %>%
count(n_positions, sort = TRUE) %>%
collect()
individual_positions is my dataset, which consists of different job positions a user held throughout her career. I tried to understand the distribution of the number of all job positions that a user ever held. And I got the following error message:
Error in `compute.arrow_dplyr_query()`:
! Invalid: Negative buffer resize: -2147483584
Backtrace:
1. ... %>% collect()
3. arrow:::collect.arrow_dplyr_query(.)
4. arrow:::compute.arrow_dplyr_query(x)
I googled what “negative buffer resize” really means, but it was in vain. Can anyone please help me with the interpretation and provide any solutions? I know it’s possible to process the dataset in SAS, but I’m an R lover and I really want to stick with it. Thanks a lot!
Important update here: Because the observations seemed to be randomly sliced into each parquet file (i.e., a user’s position-level observations may be in different parquet files), I think when performing the group_by() functions, it has to pull all parquet files together, instead of picking only the necessary few ones. This might be overwhelming for the memory. Do I have to repartition the data? Thanks!
Component(s)
R, C++
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 22 (11 by maintainers)
For anyone who had the same issue, I add
duckdb::to_duckdb()and it ran successfully now!Got it, thanks so much. I’m able to reproduce the issue.
Whoops, can you try https://www.dropbox.com/request/oAfPalVlE9utSmVMJJMj?