awkward: avro reader crash

Version of Awkward Array

ak 2.6.5 ak-cpp 34

Description and code to reproduce

I tried to read the avro file at this location https://github.com/Teradata/kylo/blob/master/samples/sample-data/avro/userdata5.avro (downloaded to local) using ak.from_avro_file and got segmentation fault. The file loads fine with fastavro.

macOS 13.6.7, conda install of python 3.10

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 20 (15 by maintainers)

Most upvoted comments

Avro-reading is not a priority, but the ForthMachine is. I didn’t want to introduce the circular dependency of running Uproot tests in the Awkward test suite, where Uproot and its tests depend on a working version of Awkward Array, but if we relax that rule, we could formally deprecate ak.from_avro and test it through Uproot instead.

These are the Uproot tests that use the ForthMachine:

# tests Uproot test file name
17 tests/test_0034_generic_objects_in_ttrees.py
1 tests/test_0053_parents_should_not_be_bases.py
1 tests/test_0126_turn_unknown_emptyarrays_into_known_types.py
1 tests/test_0442_regular_TClonesArray.py
25 tests/test_0610_awkward_form.py
65 tests/test_0637_setup_tests_for_AwkwardForth.py
4 tests/test_0798_DAOD_PHYSLITE.py
2 tests/test_0816_separate_AwkwardForth_machines_by_TBranch.py
1 tests/test_0912_fix_pandas_and_double_nested_vectors_issue_885.py
1 tests/test_1181_support_for_stl_list.py
1 tests/test_1182_add_support_for_bitset.py

especially 0637 and 0798, which were explicitly designed to test deserializations that use the ForthMachine.

I can do it

@ariostas, you found a language-level bug in AwkwardForth (as opposed to a bug in the Avro-reader, which wouldn’t worry me as much).

The reason a value popped from the stack is being interpreted as a number of items to read is because this is the implementation of vectorized read words like #i-> (to read $N$ integers from the input source, where $N$ is a number pulled from the Forth stack) and #d-> (to do the same for $N$ doubles). Not checking to see if the value from the stack is negative is a bug: AwkwardForth is not supposed to have any undefined behavior, only valid states and error states.

Most uses of num_items are in for loops like

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L2999

which shouldn’t pose a problem: if the num_items is negative, the loop is skipped just as if num_items were zero. One exception is the unusual length integer words, like #2bit->, #3bit->, #4bit->, etc. They set

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3073

and count-down items_remaining until it reaches zero:

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3078

and

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3090

These items_remaining != 0 should be changed to items_remaining > 0 to avoid an infinite loop.

However, the Avro reader couldn’t be hitting this problem because Avro doesn’t have any of these unusual (but fixed) length integers. (Avro has variable length integers, which are implemented with count < num_items.)

The one case that might cause a segfault is in the write-to-stack macros:

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3255-L3257

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3273-L3275

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3294-L3296

A negative value could break that, and it might happen in the Avro reader when a contiguous array of floating-point or boolean values are read. (The other cases are variable-length integers.)

Your fix of ensuring that num_items is nonnegative would fix that, but so would

  • changing items_remaining != 0 to items_remaining > 0 and
  • putting the if (num_items < 0) num_items = 0; inside the else { of

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3253

(I’m only saying that because this is highly streamlined code, and doing the check later would slow down fewer cases. I doubt the jump-compare-zero and assignment is much of a speedbump, though.)

So this is great! With this fix, I’d like to know if @martindurant still sees any segfault. In https://github.com/scikit-hep/awkward/issues/3188#issuecomment-2263548792, he found one with tab-complete, which can’t be related to this, but I wonder if that’s still reproducible. (It’s such an odd case, since dir doesn’t do it, but tab-complete does. Maybe there was residual left in memory…?)

I was curious about this, so I decided to look into it a bit. I’m not sure if this was already known, but I’ll document what I found here.

I was only able to reproduce the segfault with ak.from_avro_file, but not with the minimal version Jim made. The segfault is caused due to passing a negative number of elements to ForthOutputBufferOf<unsigned char>::write_uint8. Changing this line

https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L2984

to something like

num_items = stack_pop();
if (num_items < 0) num_items = 0;

makes it so that at least it crashes gracefully. As for why negative numbers end up being popped from the stack and interpreted as a length, I’m not sure. I’m not familiar with this code, so I couldn’t figure it out. Also, I’m not sure if this is related to the segfault seen from the minimal version.

If someone wants to give me a quick rundown of how the code works, I can look into this further.