arrow: [R] Segfault when collecting parquet dataset query results

Hello! I’ve been using arrow with R for a while now to great success. Recently, I’ve re-opened an old project (managed with renv, so I’m pretty confident all the package versions were the same). It is possible I upgraded the OS and/or OS packages in the meantime. Now, some of my queries on a gzip-compressed dataset of parquet files lead to a segfault:

 *** caught segfault ***
address 0x7f54ce520898, cause 'memory not mapped'

Traceback:
 1: Table__from_ExecPlanReader(self)
 2: x$read_table()
 3: as_arrow_table.RecordBatchReader(reader)
 4: as_arrow_table(reader)
 5: as_arrow_table.arrow_dplyr_query(x)
 6: as_arrow_table(x)
 7: doTryCatch(return(expr), name, parentenv, handler)
 8: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 9: tryCatchList(expr, classes, parentenv, handlers)
10: tryCatch(as_arrow_table(x), error = function(e, call = caller_env(n = 4)) {    augment_io_error_msg(e, call, schema = schema())})
11: compute.arrow_dplyr_query(x)
12: collect.arrow_dplyr_query(.)
13: collect(.)
14: d_redacted %>% group_by(year, month, cid) %>% summarize(n = n()) %>%     collect()

I have a core dump from that session, but it’s 46GB. The machine has 256GB RAM and another 256GB swap, so I’m confident that’s not the problem.

I’m not a professional in analyzing these things, but this is what I got:

Core was generated by `/usr/lib/R/bin/exec/R'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f612d4ea3b0 in arrow::compute::KeyCompare::CompareBinaryColumnToRow_avx2(bool, unsigned int, unsigned int, unsigned short const*, unsigned int const*, arrow::compute::LightContext*, arrow::compute::KeyColumnArray const&, arrow::compute::RowTableImpl const&, unsigned char*) () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
[Current thread is 1 (Thread 0x7f6093fff640 (LWP 2273813))]
(gdb) bt
#0  0x00007f612d4ea3b0 in arrow::compute::KeyCompare::CompareBinaryColumnToRow_avx2(bool, unsigned int, unsigned int, unsigned short const*, unsigned int const*, arrow::compute::LightContext*, arrow::compute::KeyColumnArray const&, arrow::compute::RowTableImpl const&, unsigned char*) () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#1  0x00007f612d4d7093 in void arrow::compute::KeyCompare::CompareBinaryColumnToRow<true>(unsigned int, unsigned int, unsigned short const*, unsigned int const*, arrow::compute::LightContext*, arrow::compute::KeyColumnArray const&, arrow::compute::RowTableImpl const&, unsigned char*) () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#2  0x00007f612d4d6278 in arrow::compute::KeyCompare::CompareColumnsToRows(unsigned int, unsigned short const*, unsigned int const*, arrow::compute::LightContext*, unsigned int*, unsigned short*, std::vector<arrow::compute::KeyColumnArray, std::allocator<arrow::compute::KeyColumnArray> > const&, arrow::compute::RowTableImpl const&, bool, unsigned char*) ()
   from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#3  0x00007f612d4d896e in ?? () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#4  0x00007f612d3a98e6 in ?? () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#5  0x00007f612d3ab154 in arrow::compute::SwissTable::find(int, unsigned int const*, unsigned char*, unsigned char const*, unsigned int*, arrow::util::TempVectorStack*, std::function<void (int, unsigned short const*, unsigned int const*, unsigned int*, unsigned short*, void*)> const&, void*) const ()
   from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#6  0x00007f612d4df2d0 in ?? () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#7  0x00007f612d4dfb73 in ?? () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#8  0x00007f612cf8da83 in arrow::acero::aggregate::GroupByNode::Merge() () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#9  0x00007f612cf8f8a3 in arrow::acero::aggregate::GroupByNode::OutputResult(bool) ()
   from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#10 0x00007f612cf941f6 in arrow::acero::aggregate::GroupByNode::InputReceived(arrow::acero::ExecNode*, arrow::compute::ExecBatch) ()
   from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#11 0x00007f612cef3f1b in arrow::acero::MapNode::InputReceived(arrow::acero::ExecNode*, arrow::compute::ExecBatch) ()
   from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#12 0x00007f612cf25dd2 in ?? () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#13 0x00007f612cf05a7e in arrow::internal::FnOnce<void ()>::FnImpl<std::_Bind<arrow::detail::ContinueFuture (arrow::Future<arrow::internal::Empty>, std::function<arrow::Status ()>)> >::invoke() ()
   from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#14 0x00007f612d290a9d in ?? () from /home/leo/.cache/R/renv/cache/v5/R-4.3/x86_64-pc-linux-gnu/arrow/15.0.1/85c24dd7844977e4a680ba28f576125c/arrow/libs/arrow.so
#15 0x00007f6136f87253 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#16 0x00007f61396a9ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#17 0x00007f613973b850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

I’ve tried:

  • Updating all the dependencies. I’m now at 15.0.1 from RSPM. Above crash is from this version.
  • Re-writing the dataset. The raw data is a bunch of CSV files, which I read -> mutate -> write to parquet
  • Checking if simple queries (dataset %>% summarize(n=n())) work, which they do

Specifically, this query works:

d_redacted %>% group_by(year, month) %>% summarize(n=n()) %>% collect()

and this doesn’t:

d_redacted %>% group_by(year, month, cid) %>% summarize(n=n()) %>% collect()

The dataset looks like this:

> d_redacted
FileSystemDataset with 1342 Parquet files
peer: string
address: string
asn: string
geolocation: string
cid: string
entry_type: string
date: date32[day]
monitor: string
year: int32
month: int32

It’s 3GB on disk, gzip compressed.

Unfortunately, I cannot share the dataset publicly as it contains sensitive information.

Overall, pretty lost now. The system is running Ubuntu 22.04, kernel:

5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Hope that helps somehow…

Component(s)

R

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (12 by maintainers)

Commits related to this issue

Most upvoted comments

Awesome work @zanmato1984, thank you for figuring this out.

The bug is that in this line: https://github.com/apache/arrow/blob/69e8a78c018da88b60f9eb2b3b45703f81f3c93d/cpp/src/arrow/compute/row/compare_internal_avx2.cc#L284 If a slot of offset_right contains a value >= 0x80000000, which is an offset in row bigger than 2GB, then it is added to right_base as a negative integer, causing gathering data from an invalid address.

Proval followed: Similar to @amoeba 's reproducing, mine is:

fault address: 0x4a7f85638
right_base: 0x0000000527e1e800
offset_right: (400023873834003288, 400025248223538328, 400025523922057392, -9217058400476779112)

Further decoding each slot of offset_right, it is:

(0x58D2B58 0x58D2B98 0x58D2C98 0x58D2CD8 0x3676B8B0 0x58D2D18 0x58D2D98 0x80166E38)

Note that the last offset is larger than 0x80000000 (which is legit, as non-avx2 version runs it right), and its signed interpretation is -2146013640. And right_base(0x0000000527e1e800) + (-2146013640) = 0x4a7f85638 is exactly the offending address. I didn’t calculate @amoeba 's case but I believe it has the same math.

I’m working on a fix.

I’ve created a reproducer and uploaded it to dropbox, thanks! The archive contains the entire RStudio project, including the renv/ subdirectory and whatnot.

When you open the project, renv will complain about broken symlinks into the cache (which is local to my user). You can probably repair that with renv::repair() or renv::restore() or something like that.

Comment/uncomment the environement-setting line to trigger the bug.

Good luck and thanks for looking into this! Let me know if you need any additional info or resources.

Edit: I uploaded the core dump and apport report as well. The latter also contains the core dump, plus some additional info, but uses whatever format apport produces…