ctypesgen: String seems incorrect

I ran into some problems, and need help with them (version 1.0.1-1.0.2 on python 3.7.4):

  File "/home/alanr/monitor/src/cma/AssimCtypes.py", line 1323, in <module>
    proj_class_new.argtypes = [gsize, String]
TypeError: item 2 in _argtypes_ passes a union by value, which is unsupported

The C source in question looks like this:

gpointer
proj_class_new(gsize objsize, ///< Size of object to be allocated
       const char * static_classname) ///< Static string giving name of class

The corresponding header file looks like this: WINEXPORT gpointer proj_class_new(gsize objsize, const char * static_classname);

The second argument is clearly a pointer, not a call by value - of anything. The generated Python looks like this:

1320 # /home/alanr/monitor/src/include/proj_classes.h: 28
1321 if hasattr(_libs['libassimilationclientlib.so'], 'proj_class_new'):
1322     proj_class_new = _libs['libassimilationclientlib.so'].proj_class_new
1323     proj_class_new.argtypes = [gsize, String]
1324     proj_class_new.restype = gpointer

String is a generated class, which looks like this:

339 class String(MutableString, Union):
340 
341     _fields_ = [("raw", POINTER(c_char)), ("data", c_char_p)]
342 
343     def __init__(self, obj=""):
344         if isinstance(obj, (bytes, UserString)):
345             self.data = bytes(obj)
346         else:
347             self.raw = obj

As an experiment, I removed the Union from the String definition, and then it seemed to work a bit better…

A few milliseconds later, I got this error: File “/home/alanr/monitor/src/cma/AssimCtypes.py”, line 1028, in <module> (‘priv’, POINTER(GSourcePrivate)), TypeError: second item in fields tuple (index 11) must be a C type

The code in question looks like this:

1015 struct__GSource._fields_ = [
1016     ('callback_data', gpointer),
1017     ('callback_funcs', POINTER(GSourceCallbackFuncs)),
1018     ('source_funcs', POINTER(GSourceFuncs)),
1019     ('ref_count', guint),
1020     ('context', POINTER(GMainContext)),
1021     ('priority', gint),
1022     ('flags', guint),
1023     ('source_id', guint),
1024     ('poll_fds', POINTER(GSList)),
1025     ('prev', POINTER(GSource)),
1026     ('next', POINTER(GSource)),
1027     ('name', String),
1028     ('priv', POINTER(GSourcePrivate)),
1029 ]

It appears that String is not a legal field type. So, this is a problem…

– Alan

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 21 (1 by maintainers)

Commits related to this issue

Locked in on Python 3.6 for release 2.0.x. This is because ctypesgen generates Python code which Python 3.7.x (for x greater than some value), and Python 3.8 doesn't think works correctly. You can re... — committed to assimilation/assimilation-official by Alan-R 5 years ago
bpo-16575: Add checks for unions passed by value to functions. (GH-16799) — committed to python/cpython by vsajip 5 years ago
Downgrade Python to avoid https://github.com/davidjamesca/ctypesgen/issues/77 — committed to BlockSigner/wowpng by betatim 4 years ago
Changes to allow running under Python 3.6.9. The capture_output paramter was introduced in Python 3.7. There is a PPA for Ubuntu 18 that allows 3.7.9 to be installed, however this has other issues d... — committed to dazzag24/wowpng by dazzag24 4 years ago

Most upvoted comments

Strange. Just tried this with Py 3.7.5 and it seems to have had no problems, including the return value being mutable as expected.

On the other hand: Any thoughts on replacing this interface with something less wonky?

I propose the following changes:

replace return values by the object above “Ustr” such that the return value is a much more simple ctypes object referring to the pointer (and the memory referenced by it) returned by a function
Whenever a “const char *” occurs, don’t do anything special–just define the interface as c_char_p and let ctypes handle the autoconversion. This implies that only bytes will be allowed (under Py3), and str must be encoded to be passed in as a parameter value. Likewise, all “const char *” return values will come back as bytes in Py3 and will have to be decoded to get to a str object.
Whenever a mutable “char *” is used as an argument, replace the argument type with something more native to ctypes and much less involved than the current string class: ctypes.POINTER(ctypes.c_char). This helps the user know they have to pass in something that resolves to a “char *”–there are several things that ctypes will automatically understand as a mutable “char *”

These changes should make the generated code:

More robust and forward compatible
- Makes the use easier because it is closer to ctypes normal use
- Avoids keeping around our own string implementation that has to be maintained
Use the automatic type conversion already done in ctypes rather than introducing our own complicated string representation
Does imply some backwards incompatibilities:
- Users will now have to explicitly send in bytes in Py3 (Py2 is unaffected here),
- Users will have to explicitly type cast the results of any functions that return mutable strings (to str or bytes if that is what the user needs–unless the user wants access to the mutable char array).
- All char buffers passed in as mutable char * parameters will need to be specifically allocated by the user (via routines such as ctypes.create_string_buffer() or perhaps directly by (ctypes.c_char*LEN)()). (Where memory is concerned here for mutable parameters, I much prefer this approach.)

Update: In case this isn’t already apparent, I was hoping for a comment from at least some ctypesgen devs/users: @Alan-R , @neteler , @betatim , @dazzag24 , …

olsonse on Mar 1, 2020

The GRASS GIS project is using ctypesgen for a long time (thanks!) but now facing problems as it fails with Python 3.7.6+.

While we could work-around it would be nice to see this fixed here. Please see the related bug report:

https://trac.osgeo.org/grass/ticket/4018

neteler on Jan 12, 2020