arrow: [C++][Python] Pyarrow.Table.join() breaks on large tables v.18.0.0.dev486

Describe the bug, including details regarding any error messages, version, and platform.

Hi, In my task I need to join two tables. One of 18m rows and another of 487m rows. t_18m.join(t_487m, keys=[''col1, 'col2', 'col3'], join_type="left outer") I was using the most actual pyarrow version which is 17 at the moment. While performing a join it breaks with Segmentation fault (core dumped) I tried to investigate and found the most recent version w/o such behaviour is v13. But I instead of Segmentation fault it: either silently produces wrong result or breaks with ! Invalid: There are more than 2^32 bytes of key data. Acero cannot process a join of this magnitude Next I searched on the github issues and found there are many similar user cases around. That’s why I didn’t include to many details in this report. You probably know these issues well.

There was #43495 the enchancement request, which as far as I understand has been included to the v18. I installed the v.18.0.0.dev486 package in order to test, but unfortunately It still throws the segmentation fault error on my case.

So if the enchancement is already merged into v.18.00 it still does not fix the problem.

Component(s)

Python

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (15 by maintainers)

Commits related to this issue

Most upvoted comments

I can now confirm that the problem does exist.

By applying filter and sum on the join result, I found my previous non-segfault runs were false positive:

    join = small.join(large, keys=['ID_DEV_STYLECOLOR_SIZE', 'ID_DEPARTMENT', 'ID_COLLECTION'], join_type='left outer')
    print("join size: {0}".format(join.num_rows))
    cond = pc.and_(pc.equal(join['ID_DEV_STYLECOLOR_SIZE'], 88506230299), pc.equal(join['ID_DEPARTMENT'], 16556030299))
    filtered = join.filter(cond)
    print("filtered")
    print(filtered)
    sum = pc.sum(join['PL_VALUE'])
    print("sum")
    print(sum)

Result:

filtered: PL_VALUE: [[null]]
...
sum: 33609597 # Another run emits 33609997

A side note: If I use right join (and switch the small/large sides), the filter/sum results are as expected.

And I also happen to have access to a x86 Ubuntu desktop, on which I reproduced the segfault.

I’m now digging into it.

Also, considering the silent wrong answer on some platforms, I’m marking this issue critical.

Thanks alot @kolfild26 for helping me to reproduce the issue!

Just added it, thanks. I also added the Critical Fix to the issue here to match the PR (we prefer to label issues). Thanks for bringing it up.

No, it’s in 20.0.0 which will come in the next three month. You can test it on a nightly build though.

Thank you so much for the explanation!

This makes sense, and I am already seeing memory and runtime savings when dealing with large tables.

Kind regards.

Ah, no worry, non-taken 😃

I can elaborate a bit about the implementation details. The join operation is implemented by “hash join” algorithm, that is, 1) “build” a hash table using the table from one side of the join (the “build side”), 2) lookup the hash table for the table from the other side (the “probe side”). We always choose the right side as the build side, that is, big in small.join(big, "left outer"), or small in big.join(small, "right outer"). Using a big table to build the hash table is inefficient, and our implementation is error-prone, so switching the sides (and the join type) will workaround the issue. And of course, the result is identical to the original join - otherwise it wouldn’t be a valid workaround 😃

Hi @kolfild26 , thanks for reporting this.

There are lots of solved issues from v13 to v18 that may cause silent wrong answer or segfault in hash join, and possibly more unrevealed ones as well. So it is not too surprising that different versions behave differently.

Could you please provide us the complete schemas and the estimated sizes of both tables? And better yet, could you give a more-or-less working limit of your case? These are essential informations to investigate this issue.

Also, there might be a workaround that worth a try, change t_18m.join(t_487m, keys=[''col1, 'col2', 'col3'], join_type="left outer") to t_487m.join(t_18m, keys=[''col1, 'col2', 'col3'], join_type="right outer"). (I assume t_18m is much smaller than t_487m and this will make our hash join to use the small table to build the hash table.)

Thanks.