arrow: [R][C++] "negative buffer resize" error with arrow and dplyr in R

Hi everyone,

I was working on a large dataset with over 1 billion observations, stored in 3040 parquet files, with 41 variables. I read the data with open_dataset() and then wanted to apply dplyr functions:

individual_positions %>% 
  group_by(user_id) %>% 
  summarize(n_positions = n()) %>% 
  count(n_positions, sort = TRUE) %>% 
  collect()

individual_positions is my dataset, which consists of different job positions a user held throughout her career. I tried to understand the distribution of the number of all job positions that a user ever held. And I got the following error message:

Error in `compute.arrow_dplyr_query()`:
! Invalid: Negative buffer resize: -2147483584
Backtrace:
 1. ... %>% collect()
 3. arrow:::collect.arrow_dplyr_query(.)
 4. arrow:::compute.arrow_dplyr_query(x)

I googled what “negative buffer resize” really means, but it was in vain. Can anyone please help me with the interpretation and provide any solutions? I know it’s possible to process the dataset in SAS, but I’m an R lover and I really want to stick with it. Thanks a lot!

Important update here: Because the observations seemed to be randomly sliced into each parquet file (i.e., a user’s position-level observations may be in different parquet files), I think when performing the group_by() functions, it has to pull all parquet files together, instead of picking only the necessary few ones. This might be overwhelming for the memory. Do I have to repartition the data? Thanks!

Component(s)

R, C++

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 22 (11 by maintainers)

Most upvoted comments

For anyone who had the same issue, I add duckdb::to_duckdb() and it ran successfully now!

individual_positions %>% 
  to_duckdb() %>%
  group_by(user_id) %>% 
  summarize(n_positions = n()) %>% 
  count(n_positions, sort = TRUE) %>% 
  collect()

Got it, thanks so much. I’m able to reproduce the issue.