ctypesgen: Incorrect handling of flexible array members (FAM) leads to wrong structure sizes and offsets
Description
When processing C structures that contain flexible array members, ctypesgen incorrectly calculates structure sizes and field offsets.
This leads to incompatible memory layouts between C and Python code, making it impossible to correctly interact with C structures that use FAMs.
The main issues are:
- FAMs are treated as fixed-size arrays (8 bytes) instead of zero-sized arrays
- Field offsets after FAMs are miscalculated
Test Case
To reproduce the issue, use the following files:
test_structs.h:
#ifndef TEST_STRUCTS_H
#define TEST_STRUCTS_H
#include <stdint.h>
// Header struct with flexible array member
struct msg_header {
uint32_t size; // Total size of the message
uint8_t type; // Type of the message
uint8_t payload[]; // Flexible array member
};
// First message type
struct msg_type1 {
struct msg_header header;
uint32_t value1;
uint32_t value2;
uint8_t extra[]; // Flexible array member for additional data
};
// Second message type
struct msg_type2 {
struct msg_header header;
uint64_t timestamp;
uint8_t extra[]; // Flexible array member for additional data
};
// Union to access either just the header or the complete message
union message {
struct msg_header header;
struct msg_type1 type1;
struct msg_type2 type2;
};
#endif // TEST_STRUCTS_H
C program to show correct offsets (struct_offsets.c):
#include <stdio.h>
#include <stddef.h>
#include "test_structs.h"
#define PRINT_OFFSET(type, field) \
printf("%-20s %-20s offset: %3lu size: %3lu\n", #type, #field, offsetof(type, field), sizeof(((type*)0)->field))
#define PRINT_TOTAL_SIZE(type) \
printf("%-20s %-20s size: %3lu\n", #type, "total", sizeof(type))
int main() {
printf("=== Structure Offsets and Sizes ===\n\n");
// msg_header offsets
printf("msg_header:\n");
PRINT_OFFSET(struct msg_header, size);
PRINT_OFFSET(struct msg_header, type);
printf("%-20s %-20s offset: %3lu size: FAM\n", "struct msg_header", "payload",
offsetof(struct msg_header, payload));
PRINT_TOTAL_SIZE(struct msg_header);
printf("\n");
// msg_type1 offsets
printf("msg_type1:\n");
PRINT_OFFSET(struct msg_type1, header);
PRINT_OFFSET(struct msg_type1, value1);
PRINT_OFFSET(struct msg_type1, value2);
printf("%-20s %-20s offset: %3lu size: FAM\n", "struct msg_type1", "extra",
offsetof(struct msg_type1, extra));
PRINT_TOTAL_SIZE(struct msg_type1);
printf("\n");
// msg_type2 offsets
printf("msg_type2:\n");
PRINT_OFFSET(struct msg_type2, header);
PRINT_OFFSET(struct msg_type2, timestamp);
printf("%-20s %-20s offset: %3lu size: FAM\n", "struct msg_type2", "extra",
offsetof(struct msg_type2, extra));
PRINT_TOTAL_SIZE(struct msg_type2);
printf("\n");
// union message size
printf("union message:\n");
PRINT_TOTAL_SIZE(union message);
return 0;
}
Python program using ctypesgen bindings (python_offsets.py):
import ctypes
from ctypesgen_test import *
def get_field_offset_and_size(struct_type, field_name):
"""Get the offset and size of a field using ctypes internals"""
for field_tuple in struct_type._fields_:
if field_tuple[0] == field_name:
field_type = field_tuple[1]
offset = getattr(struct_type, field_name).offset
try:
size = ctypes.sizeof(field_type)
except TypeError:
size = 0
return offset, size
return None, None
def print_offset(struct_type, field_name):
"""Print offset and size of a struct field"""
offset, size = get_field_offset_and_size(struct_type, field_name)
if offset is not None:
if size > 0:
print(f"{struct_type.__name__:<20} {field_name:<20} offset: {offset:3d} size: {size:3d}")
else:
print(f"{struct_type.__name__:<20} {field_name:<20} offset: {offset:3d} size: FAM")
else:
print(f"{struct_type.__name__:<20} {field_name:<20} Error: Field not found")
def print_total_size(struct_type):
"""Print total size of a struct"""
try:
size = ctypes.sizeof(struct_type)
print(f"{struct_type.__name__:<20} {'total':<20} size: {size:3d}")
except Exception as e:
print(f"{struct_type.__name__:<20} {'total':<20} Error: {str(e)}")
def main():
print("=== Structure Offsets and Sizes (Python/ctypesgen) ===\n")
print("msg_header:")
print_offset(msg_header, 'size')
print_offset(msg_header, 'type')
print_offset(msg_header, 'payload')
print_total_size(msg_header)
print()
print("msg_type1:")
print_offset(msg_type1, 'header')
print_offset(msg_type1, 'value1')
print_offset(msg_type1, 'value2')
print_offset(msg_type1, 'extra')
print_total_size(msg_type1)
print()
print("msg_type2:")
print_offset(msg_type2, 'header')
print_offset(msg_type2, 'timestamp')
print_offset(msg_type2, 'extra')
print_total_size(msg_type2)
print()
print("union message:")
print_total_size(message)
if __name__ == "__main__":
main()
Steps to Reproduce
- Save the header file as
test_structs.h - Compile and run the C program:
gcc -o struct_offsets struct_offsets.c ./struct_offsets - Generate Python bindings and run the Python program:
ctypesgen test_structs.h -o ctypesgen_test.py python3 python_offsets.py
Current Output
C Program (correct):
msg_header:
struct msg_header size offset: 0 size: 4
struct msg_header type offset: 4 size: 1
struct msg_header payload offset: 5 size: FAM
struct msg_header total size: 8
msg_type1:
struct msg_type1 header offset: 0 size: 8
struct msg_type1 value1 offset: 8 size: 4
struct msg_type1 value2 offset: 12 size: 4
struct msg_type1 extra offset: 16 size: FAM
struct msg_type1 total size: 16
msg_type2:
struct msg_type2 header offset: 0 size: 8
struct msg_type2 timestamp offset: 8 size: 8
struct msg_type2 extra offset: 16 size: FAM
struct msg_type2 total size: 16
union message:
union message total size: 16
Python/ctypesgen (incorrect):
msg_header:
struct_msg_header size offset: 0 size: 4
struct_msg_header type offset: 4 size: 1
struct_msg_header payload offset: 8 size: 8
struct_msg_header total size: 16
msg_type1:
struct_msg_type1 header offset: 0 size: 16
struct_msg_type1 value1 offset: 16 size: 4
struct_msg_type1 value2 offset: 20 size: 4
struct_msg_type1 extra offset: 24 size: 8
struct_msg_type1 total size: 32
msg_type2:
struct_msg_type2 header offset: 0 size: 16
struct_msg_type2 timestamp offset: 16 size: 8
struct_msg_type2 extra offset: 24 size: 8
struct_msg_type2 total size: 32
union message:
union_message total size: 32
Expected Behavior
The Python/ctypesgen output should match the C output in terms of:
- Structure sizes
- Field offsets
- Flexible array member handling
Suggested Fix
The ctypesgen code needs to be modified to:
- Correctly identify flexible array members in C structures
- Generate Python/
ctypescode that represents FAMs with zero size - Properly calculate structure sizes and field offsets when FAMs are present
- Handle nested structures with FAMs correctly
- Ensure union sizes are calculated correctly when they contain structures with FAMs
Environment
- Operating System: Linux (but issue is platform-independent)
- Python Version: 3.8+
ctypesgenversion: 1.1.1
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 1
- Comments: 37 (24 by maintainers)
Commits related to this issue
- Handle FAM at end of struct as zero-sized array Thanks to @Natanel-Shitrit CC https://github.com/ctypesgen/ctypesgen/issues/219 — committed to pypdfium2-team/ctypesgen by mara004 2 years ago
- Handle FAM at end of struct as zero-sized array (#19) Thanks to @Natanel-Shitrit CC https://github.com/ctypesgen/ctypesgen/issues/219 — committed to pypdfium2-team/ctypesgen by mara004 2 years ago
@mara004 Thank you very much!
I’ve tested your fork and the changes works great!
@Natanel-Shitrit I have improved the change a bit, added a test case, and merged this at the fork. See https://patch-diff.githubusercontent.com/raw/pypdfium2-team/ctypesgen/pull/19.diff for the final patch.
How about this?
Yep it should be inside the
forloop right below it. Pasted it in the wrong place 😅Edit: now fixed 😃
Fixed it!
New patch
https://github.com/ctypesgen/ctypesgen/blob/554faced3d41415c91226f62961cb5b9825fe7eb/ctypesgen/parser/ctypesparser.py#L94-L99
This ensures that it will only affect structs because I’ve moved it to the
make_struct_from_specifiermethod.Test Case:
code:
ctypesgenoutput:I’ll create a PR with this change + test cases tomorrow.
I just tested a function declaration with array with your patch. Unfortunately it does also affect (and thus break) functions indeed:
Alright, still thanks for all the help so far 😄
I agree this is far from a high priority issue, but it would still be nice to have
ctypesgenoutput the correct structs (just to behave the same as the C program).Yes, but (I guess) this was written while thinking about function arguments (and not structs) since you cannot define an empty sized array in a struct without it being a FAM.
Anyway, I hope someone who knows a bit more about the implications of this patch will join the conversation so we can be sure this will not break anything else.
In practice, yes, I suppose you can argue that way. 😃 However, that’s not a reason not to do it properly when possible, so I would still argue that
(c_uint8 * obj.size).from_address(addressof(obj.payload))is cleaner thancast(addressof(obj.payload), POINTER(c_uint8*obj.size)).contents.@TheCodeArtist
That’s because it thinks it’s a pointer, yes.
Thanks for the workaround, but unfortunately the code I work with is not under my control - I cannot modify it.
Just for reference, I used trolldbois/ctypeslib which uses LLVM clang, and it correctly identified the FAMs and creates a “zero-sized array” field.
@mara004
The current struct is as seen above, and the expected struct could be something similar to what @TheCodeArtist created with the explicit zero-sized array:
General question to anyone: Is it intentional that an array without an explicit size is interpreted as a pointer or is it a missed edge case in the parser?
That would be determined at runtime, yes. In the example test case I provided, the
sizefield is the total size of the struct, which means you can calculate the size of the array.I don’t expect the parser to give a size to the array at the parsing stage because that’s impossible. We can just give it a size of “nothing” and delegate the task to the user to calculate the size according to how it was stored, because there is no one way to do that, the user should calculate it himself according to how it was designed.
I can imagine why you see this as a nested struct (with a FAM field) but in this case the FAM covers all the data that comes after that - it doesn’t store extra data but the nested struct itself, up to
size.Just to mention, I agree this is a hack-ish (and somewhat flawed) design, and I would never create something like this myself, and wish I didn’t had to work with it 😅
Is this because a reference to a C-array is being treated as a pointer? 🤔 (or is that 8bytes on a 64-bit system just a coincidence?)
As a quick experiment, explicitly specifying a zero-length array seems to do the trick.
…and subsequently appears to be triggering the expected behavior
@Natanel-Shitrit is this acceptable?
Also, according to https://stackoverflow.com/questions/29354780/flexible-array-member-in-a-nested-struct, a FAM can only be at the end of a struct, and nested structs with intermediate FAMs are not valid per the C standard ?
To get a better perspective of this, could you make a comparison of the current output for the struct(s), and the proper output you would expect instead? i.e. which ctypes param(s) should we set that we currently don’t?
I’m not an expert, but I came across https://stackoverflow.com/questions/63853622/python-ctypes-struct-with-flexible-array-member-to-bytearray, and the way I understand Mark Tolonen’s answer is that we’d need to know the concrete array length, which only the caller does (in a use case specific way), not the parser ?