ctypesgen: Incorrect handling of flexible array members (FAM) leads to wrong structure sizes and offsets

Description

When processing C structures that contain flexible array members, ctypesgen incorrectly calculates structure sizes and field offsets. This leads to incompatible memory layouts between C and Python code, making it impossible to correctly interact with C structures that use FAMs.

The main issues are:

  • FAMs are treated as fixed-size arrays (8 bytes) instead of zero-sized arrays
  • Field offsets after FAMs are miscalculated

Test Case

To reproduce the issue, use the following files:

test_structs.h:

#ifndef TEST_STRUCTS_H
#define TEST_STRUCTS_H

#include <stdint.h>

// Header struct with flexible array member
struct msg_header {
    uint32_t size;        // Total size of the message
    uint8_t type;         // Type of the message
    uint8_t payload[];    // Flexible array member
};

// First message type
struct msg_type1 {
    struct msg_header header;
    uint32_t value1;
    uint32_t value2;
    uint8_t extra[];      // Flexible array member for additional data
};

// Second message type
struct msg_type2 {
    struct msg_header header;
    uint64_t timestamp;
    uint8_t extra[];      // Flexible array member for additional data
};

// Union to access either just the header or the complete message
union message {
    struct msg_header header;
    struct msg_type1 type1;
    struct msg_type2 type2;
};

#endif // TEST_STRUCTS_H

C program to show correct offsets (struct_offsets.c):

#include <stdio.h>
#include <stddef.h>
#include "test_structs.h"

#define PRINT_OFFSET(type, field) \
    printf("%-20s %-20s offset: %3lu size: %3lu\n", #type, #field, offsetof(type, field), sizeof(((type*)0)->field))

#define PRINT_TOTAL_SIZE(type) \
    printf("%-20s %-20s size:  %3lu\n", #type, "total", sizeof(type))

int main() {
    printf("=== Structure Offsets and Sizes ===\n\n");
    
    // msg_header offsets
    printf("msg_header:\n");
    PRINT_OFFSET(struct msg_header, size);
    PRINT_OFFSET(struct msg_header, type);
    printf("%-20s %-20s offset: %3lu size:  FAM\n", "struct msg_header", "payload", 
           offsetof(struct msg_header, payload));
    PRINT_TOTAL_SIZE(struct msg_header);
    printf("\n");

    // msg_type1 offsets
    printf("msg_type1:\n");
    PRINT_OFFSET(struct msg_type1, header);
    PRINT_OFFSET(struct msg_type1, value1);
    PRINT_OFFSET(struct msg_type1, value2);
    printf("%-20s %-20s offset: %3lu size:  FAM\n", "struct msg_type1", "extra",
           offsetof(struct msg_type1, extra));
    PRINT_TOTAL_SIZE(struct msg_type1);
    printf("\n");

    // msg_type2 offsets
    printf("msg_type2:\n");
    PRINT_OFFSET(struct msg_type2, header);
    PRINT_OFFSET(struct msg_type2, timestamp);
    printf("%-20s %-20s offset: %3lu size:  FAM\n", "struct msg_type2", "extra",
           offsetof(struct msg_type2, extra));
    PRINT_TOTAL_SIZE(struct msg_type2);
    printf("\n");

    // union message size
    printf("union message:\n");
    PRINT_TOTAL_SIZE(union message);
    
    return 0;
}

Python program using ctypesgen bindings (python_offsets.py):

import ctypes
from ctypesgen_test import *

def get_field_offset_and_size(struct_type, field_name):
    """Get the offset and size of a field using ctypes internals"""
    for field_tuple in struct_type._fields_:
        if field_tuple[0] == field_name:
            field_type = field_tuple[1]
            offset = getattr(struct_type, field_name).offset
            try:
                size = ctypes.sizeof(field_type)
            except TypeError:
                size = 0
            return offset, size
    return None, None

def print_offset(struct_type, field_name):
    """Print offset and size of a struct field"""
    offset, size = get_field_offset_and_size(struct_type, field_name)
    
    if offset is not None:
        if size > 0:
            print(f"{struct_type.__name__:<20} {field_name:<20} offset: {offset:3d} size: {size:3d}")
        else:
            print(f"{struct_type.__name__:<20} {field_name:<20} offset: {offset:3d} size: FAM")
    else:
        print(f"{struct_type.__name__:<20} {field_name:<20} Error: Field not found")

def print_total_size(struct_type):
    """Print total size of a struct"""
    try:
        size = ctypes.sizeof(struct_type)
        print(f"{struct_type.__name__:<20} {'total':<20} size:  {size:3d}")
    except Exception as e:
        print(f"{struct_type.__name__:<20} {'total':<20} Error: {str(e)}")

def main():
    print("=== Structure Offsets and Sizes (Python/ctypesgen) ===\n")
    
    print("msg_header:")
    print_offset(msg_header, 'size')
    print_offset(msg_header, 'type')
    print_offset(msg_header, 'payload')
    print_total_size(msg_header)
    print()

    print("msg_type1:")
    print_offset(msg_type1, 'header')
    print_offset(msg_type1, 'value1')
    print_offset(msg_type1, 'value2')
    print_offset(msg_type1, 'extra')
    print_total_size(msg_type1)
    print()

    print("msg_type2:")
    print_offset(msg_type2, 'header')
    print_offset(msg_type2, 'timestamp')
    print_offset(msg_type2, 'extra')
    print_total_size(msg_type2)
    print()

    print("union message:")
    print_total_size(message)

if __name__ == "__main__":
    main()

Steps to Reproduce

  1. Save the header file as test_structs.h
  2. Compile and run the C program:
    gcc -o struct_offsets struct_offsets.c
    ./struct_offsets
    
  3. Generate Python bindings and run the Python program:
    ctypesgen test_structs.h -o ctypesgen_test.py
    python3 python_offsets.py
    

Current Output

C Program (correct):

msg_header:
struct msg_header    size                 offset:   0 size:   4
struct msg_header    type                 offset:   4 size:   1
struct msg_header    payload              offset:   5 size:  FAM
struct msg_header    total                size:    8

msg_type1:
struct msg_type1     header               offset:   0 size:   8
struct msg_type1     value1               offset:   8 size:   4
struct msg_type1     value2               offset:  12 size:   4
struct msg_type1     extra                offset:  16 size:  FAM
struct msg_type1     total                size:   16

msg_type2:
struct msg_type2     header               offset:   0 size:   8
struct msg_type2     timestamp            offset:   8 size:   8
struct msg_type2     extra                offset:  16 size:  FAM
struct msg_type2     total                size:   16

union message:
union message        total                size:   16

Python/ctypesgen (incorrect):

msg_header:
struct_msg_header    size                 offset:   0 size:   4
struct_msg_header    type                 offset:   4 size:   1
struct_msg_header    payload              offset:   8 size:   8
struct_msg_header    total                size:   16

msg_type1:
struct_msg_type1     header               offset:   0 size:  16
struct_msg_type1     value1               offset:  16 size:   4
struct_msg_type1     value2               offset:  20 size:   4
struct_msg_type1     extra                offset:  24 size:   8
struct_msg_type1     total                size:   32

msg_type2:
struct_msg_type2     header               offset:   0 size:  16
struct_msg_type2     timestamp            offset:  16 size:   8
struct_msg_type2     extra                offset:  24 size:   8
struct_msg_type2     total                size:   32

union message:
union_message        total                size:   32

Expected Behavior

The Python/ctypesgen output should match the C output in terms of:

  1. Structure sizes
  2. Field offsets
  3. Flexible array member handling

Suggested Fix

The ctypesgen code needs to be modified to:

  1. Correctly identify flexible array members in C structures
  2. Generate Python/ctypes code that represents FAMs with zero size
  3. Properly calculate structure sizes and field offsets when FAMs are present
  4. Handle nested structures with FAMs correctly
  5. Ensure union sizes are calculated correctly when they contain structures with FAMs

Environment

  • Operating System: Linux (but issue is platform-independent)
  • Python Version: 3.8+
  • ctypesgen version: 1.1.1

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 37 (24 by maintainers)

Commits related to this issue

Most upvoted comments

@mara004 Thank you very much!

I’ve tested your fork and the changes works great!

@Natanel-Shitrit I have improved the change a bit, added a test case, and merged this at the fork. See https://patch-diff.githubusercontent.com/raw/pypdfium2-team/ctypesgen/pull/19.diff for the final patch.

How about this?

--- a/src/ctypesgen/parser/ctypesparser.py
+++ b/src/ctypesgen/parser/ctypesparser.py
@@ -90,11 +90,17 @@ class CtypesParser(CParser):
 
         if specifier.declarations:
             members = []
+            
+            # FAM (flexible array member) at end of struct should be handled as zero-sized array (see GH issue 219)
+            last_declarator = specifier.declarations[-1].declarator
+            if last_declarator and last_declarator.array and last_declarator.array.size is None:
+                last_declarator.array.size = ConstantExpressionNode(0)
+            
             for declaration in specifier.declarations:
+                declarator = declaration.declarator
                 t = self.get_ctypes_type(
-                    declaration.type, declaration.declarator, check_qualifiers=True
+                    declaration.type, declarator, check_qualifiers=True
                 )
-                declarator = declaration.declarator
                 if declarator is None:
                     # Anonymous field in nested union/struct (C11/GCC).
                     name = None

Yep it should be inside the for loop right below it. Pasted it in the wrong place 😅

Edit: now fixed 😃

Fixed it!

New patch

https://github.com/ctypesgen/ctypesgen/blob/554faced3d41415c91226f62961cb5b9825fe7eb/ctypesgen/parser/ctypesparser.py#L94-L99

 if specifier.declarations: 
     members = []
     for declaration in specifier.declarations:  
+        if declaration.declarator.array and declaration.declarator.array.size is None:
+            declaration.declarator.array.size = ConstantExpressionNode(0)
         t = self.get_ctypes_type( 
             declaration.type, declaration.declarator, check_qualifiers=True 
         )

This ensures that it will only affect structs because I’ve moved it to the make_struct_from_specifier method.

Test Case:

code:

// test_structs.h

#ifndef TEST_STRUCTS_H
#define TEST_STRUCTS_H

#include <stdint.h>

// input
void arraytest(int a[]);

// Header struct with flexible array member
struct msg_header {
    uint32_t size;        // Total size of the message
    uint8_t type;         // Type of the message
    uint8_t payload[];    // Flexible array member
};

#endif // TEST_STRUCTS_H

ctypesgen output:

# /home/user/test_structs.h: 9
for _lib in _libs.values():
    if not _lib.has("arraytest", "cdecl"):
        continue
    arraytest = _lib.get("arraytest", "cdecl")
    arraytest.argtypes = [POINTER(c_int)]
    arraytest.restype = None
    break

# /home/user/test_structs.h: 12
class struct_msg_header(Structure):
    pass

struct_msg_header.__slots__ = [
    'size',
    'type',
    'payload',
]
struct_msg_header._fields_ = [
    ('size', uint32_t),
    ('type', uint8_t),
    ('payload', uint8_t * int(0)),
]

I’ll create a PR with this change + test cases tomorrow.

Ah yes, but then we need to make sure the code we patch really only affects structs, and not functions.

I just tested a function declaration with array with your patch. Unfortunately it does also affect (and thus break) functions indeed:

// input
void arraytest(int a[]);
# output
# ./arraytest.h: 1
if hasattr(_libs['test'], 'arraytest'):
    arraytest = _libs['test']['arraytest']
    arraytest.argtypes = [c_int * int(0)]
    arraytest.restype = None

Again, unfortunately I’m no expert concerning the parser / data collector backend (and I’m currently immersed in pypdfium2’s devel branch), so I’d like to wait for @nilason or one of the senior ctypesgen experts to confirm.

Alright, still thanks for all the help so far 😄

I don’t think this is high priority, as what that header does (nested FAM) doesn’t seem to be exactly standard-conformant, see the thread referenced in https://github.com/ctypesgen/ctypesgen/issues/219#issuecomment-2597373547.

I agree this is far from a high priority issue, but it would still be nice to have ctypesgen output the correct structs (just to behave the same as the C program).

What makes me feel a bit unsure is that treating an array with count None as pointer type seems to be sort of intentional?

Yes, but (I guess) this was written while thinking about function arguments (and not structs) since you cannot define an empty sized array in a struct without it being a FAM.

Anyway, I hope someone who knows a bit more about the implications of this patch will join the conversation so we can be sure this will not break anything else.

In practice, yes, I suppose you can argue that way. 😃 However, that’s not a reason not to do it properly when possible, so I would still argue that (c_uint8 * obj.size).from_address(addressof(obj.payload)) is cleaner than cast(addressof(obj.payload), POINTER(c_uint8*obj.size)).contents.

@TheCodeArtist

Is this because a reference to a C-array is being treated as a pointer? 🤔

That’s because it thinks it’s a pointer, yes.

struct_msg_header._fields_ = [
    ('size', uint32_t),
    ('type', uint8_t),
    ('payload', POINTER(uint8_t)),
]

Thanks for the workaround, but unfortunately the code I work with is not under my control - I cannot modify it.

Just for reference, I used trolldbois/ctypeslib which uses LLVM clang, and it correctly identified the FAMs and creates a “zero-sized array” field.

@mara004

To get a better perspective of this, could you make a comparison of the current output for the struct(s), and the proper output you would expect instead? i.e. which ctypes param(s) should we set that we currently don’t?

The current struct is as seen above, and the expected struct could be something similar to what @TheCodeArtist created with the explicit zero-sized array:

struct_msg_header._fields_ = [
    ('size', uint32_t),
    ('type', uint8_t),
    ('payload', uint8_t * int(0)),
]

General question to anyone: Is it intentional that an array without an explicit size is interpreted as a pointer or is it a missed edge case in the parser?

we’d need to know the concrete array length, which only the caller does (in a use case specific way), not the parser ?

That would be determined at runtime, yes. In the example test case I provided, the size field is the total size of the struct, which means you can calculate the size of the array.

I don’t expect the parser to give a size to the array at the parsing stage because that’s impossible. We can just give it a size of “nothing” and delegate the task to the user to calculate the size according to how it was stored, because there is no one way to do that, the user should calculate it himself according to how it was designed.

Also, according to https://stackoverflow.com/questions/29354780/flexible-array-member-in-a-nested-struct, a FAM can only be at the end of a struct, and nested structs with intermediate FAMs are not valid per the C standard ?

I can imagine why you see this as a nested struct (with a FAM field) but in this case the FAM covers all the data that comes after that - it doesn’t store extra data but the nested struct itself, up to size.

Just to mention, I agree this is a hack-ish (and somewhat flawed) design, and I would never create something like this myself, and wish I didn’t had to work with it 😅

FAMs are treated as fixed-size arrays (8 bytes) instead of zero-sized arrays

Is this because a reference to a C-array is being treated as a pointer? 🤔 (or is that 8bytes on a 64-bit system just a coincidence?)

As a quick experiment, explicitly specifying a zero-length array seems to do the trick.

-    uint8_t payload[];    // Flexible array member
+    uint8_t payload[0];    // Flexible array member

Image

…and subsequently appears to be triggering the expected behavior

python3 python_offsets.py
=== Structure Offsets and Sizes (Python/ctypesgen) ===

msg_header:
struct_msg_header    size                 offset:   0 size:   4
struct_msg_header    type                 offset:   4 size:   1
struct_msg_header    payload              offset:   5 size: FAM    <-- FAM detected properly now.
struct_msg_header    total                size:    8

msg_type1:
struct_msg_type1     header               offset:   0 size:   8
struct_msg_type1     value1               offset:   8 size:   4
struct_msg_type1     value2               offset:  12 size:   4
struct_msg_type1     extra                offset:  16 size:   8    <-- Specifying an explicit zero-length array should fix this as well.
struct_msg_type1     total                size:   24

msg_type2:
struct_msg_type2     header               offset:   0 size:   8
struct_msg_type2     timestamp            offset:   8 size:   8
struct_msg_type2     extra                offset:  16 size:   8    <-- Specifying an explicit zero-length array should fix this as well.
struct_msg_type2     total                size:   24

union message:
union_message        total                size:   24

Specifying an explicit zero-length array

@Natanel-Shitrit is this acceptable?

Also, according to https://stackoverflow.com/questions/29354780/flexible-array-member-in-a-nested-struct, a FAM can only be at the end of a struct, and nested structs with intermediate FAMs are not valid per the C standard ?

To get a better perspective of this, could you make a comparison of the current output for the struct(s), and the proper output you would expect instead? i.e. which ctypes param(s) should we set that we currently don’t?

I’m not an expert, but I came across https://stackoverflow.com/questions/63853622/python-ctypes-struct-with-flexible-array-member-to-bytearray, and the way I understand Mark Tolonen’s answer is that we’d need to know the concrete array length, which only the caller does (in a use case specific way), not the parser ?