awkward: avro reader crash
Version of Awkward Array
ak 2.6.5 ak-cpp 34
Description and code to reproduce
I tried to read the avro file at this location https://github.com/Teradata/kylo/blob/master/samples/sample-data/avro/userdata5.avro (downloaded to local) using ak.from_avro_file and got segmentation fault. The file loads fine with fastavro.
macOS 13.6.7, conda install of python 3.10
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20 (15 by maintainers)
Avro-reading is not a priority, but the ForthMachine is. I didn’t want to introduce the circular dependency of running Uproot tests in the Awkward test suite, where Uproot and its tests depend on a working version of Awkward Array, but if we relax that rule, we could formally deprecate
ak.from_avroand test it through Uproot instead.These are the Uproot tests that use the ForthMachine:
especially 0637 and 0798, which were explicitly designed to test deserializations that use the ForthMachine.
I can do it
@ariostas, you found a language-level bug in AwkwardForth (as opposed to a bug in the Avro-reader, which wouldn’t worry me as much).
The reason a value popped from the stack is being interpreted as a number of items to read is because this is the implementation of vectorized read words like
#i->(to read $N$ integers from the input source, where $N$ is a number pulled from the Forth stack) and#d->(to do the same for $N$ doubles). Not checking to see if the value from the stack is negative is a bug: AwkwardForth is not supposed to have any undefined behavior, only valid states and error states.Most uses of
num_itemsare in for loops likehttps://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L2999
which shouldn’t pose a problem: if the
num_itemsis negative, the loop is skipped just as ifnum_itemswere zero. One exception is the unusual length integer words, like#2bit->,#3bit->,#4bit->, etc. They sethttps://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3073
and count-down
items_remaininguntil it reaches zero:https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3078
and
https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3090
These
items_remaining != 0should be changed toitems_remaining > 0to avoid an infinite loop.However, the Avro reader couldn’t be hitting this problem because Avro doesn’t have any of these unusual (but fixed) length integers. (Avro has variable length integers, which are implemented with
count < num_items.)The one case that might cause a segfault is in the write-to-stack macros:
https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3255-L3257
https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3273-L3275
https://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3294-L3296
A negative value could break that, and it might happen in the Avro reader when a contiguous array of floating-point or boolean values are read. (The other cases are variable-length integers.)
Your fix of ensuring that
num_itemsis nonnegative would fix that, but so woulditems_remaining != 0toitems_remaining > 0andif (num_items < 0) num_items = 0;inside theelse {ofhttps://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L3253
(I’m only saying that because this is highly streamlined code, and doing the check later would slow down fewer cases. I doubt the jump-compare-zero and assignment is much of a speedbump, though.)
So this is great! With this fix, I’d like to know if @martindurant still sees any segfault. In https://github.com/scikit-hep/awkward/issues/3188#issuecomment-2263548792, he found one with tab-complete, which can’t be related to this, but I wonder if that’s still reproducible. (It’s such an odd case, since
dirdoesn’t do it, but tab-complete does. Maybe there was residual left in memory…?)I was curious about this, so I decided to look into it a bit. I’m not sure if this was already known, but I’ll document what I found here.
I was only able to reproduce the segfault with
ak.from_avro_file, but not with the minimal version Jim made. The segfault is caused due to passing a negative number of elements toForthOutputBufferOf<unsigned char>::write_uint8. Changing this linehttps://github.com/scikit-hep/awkward/blob/c3c4f6f8eca5ac76fb54c90874c5a499894aa1d5/awkward-cpp/src/libawkward/forth/ForthMachine.cpp#L2984
to something like
makes it so that at least it crashes gracefully. As for why negative numbers end up being popped from the stack and interpreted as a length, I’m not sure. I’m not familiar with this code, so I couldn’t figure it out. Also, I’m not sure if this is related to the segfault seen from the minimal version.
If someone wants to give me a quick rundown of how the code works, I can look into this further.