pywinauto: Cannot Import pywinauto (comtypes passes a union by value)

Happily the changes appear to have been reverted for now https://github.com/python/cpython/pull/17960

On another note…

We are talking about 1’s and 0’s here… It’s always 1’s and 0’s. Windows is all it’s glory really doesn’t give a crap what is passed to it as long as there are enough bytes allocated to fit the information needed.

I have used this kind of an approach on more then one occasion because of the Windows API being goofy. or simply because of an unknown amount of data being placed.

I recently ran into this issue when writing a pure python eHome CIR binding.

There is not a huge amount of documentation for this portion of the Windows API. and what is available conflicts with other documentation.

ULONG_PTR data type has a double meaning when using it with ctypes. some of the fields containing it. when reading what the ULONG_PTR is… it either a ULONG or a ULONGLONG depending on architecture… This is where the documentation failed… some of the fields needed to be set based on the bitness of python while others had to be set based on the bitness of windows. and others were just flat out wrong and are not ULONG’s at all.

class _IR_RECEIVE_PARAMS(ctypes.Structure):
    _fields_ = [
        ('DataEnd', ULONG_PTR),  # out
        ('ByteCount', ULONG_PTR),  # in
        ('Data', ULONG_PTR * 1),  # out
    ]

So I used code like this to deal with the problem at hand.,

class _IR_RECEIVE_PARAMS(ctypes.Structure):
    _fields_ = [
        ('DataEnd', ULONG_PTR),  # out
        ('ByteCount', ULONG_PTR),  # in
    ]
obj = _IR_RECEIVE_PARAMS()
# LONG or ULONG does not matter they are of the same size
offset = ctypes.sizeof(_IR_RECEIVE_PARAMS) + ctypes.sizeof(LONG)
size = offset + (ctypes.sizeof(LONG) * 100)
func(ctypes.byref(obj), size)

data = ctypes.cast(ctypes.byref(obj), POINTER(ctypes.c_ubyte))

It was something to this nature… I can’t exactly remember. I do know that I had found a bug in ctypes with the garbage collection of ctypes objects being passed between function calls because of having to deal with this. Now I know in this case I am passing a pointer. but windows does not really care what the data type is so long as there is enough space or any data it needs can be found at specific locations in memory. I would imagine that anything can be pass so long as it, A: is the correct size and B: data is positioned properly. if that is the case then it should be possible to do the 2 step around that exception that they have added. Now I am not saying it would be easy to do. This is a tad above my head so I really do not know how else to explain it… could lead to an idea of a fix… who knows… I thought it worth mentioning.

well see that’s all fine and dandy… But no matter what change you make to the thing I think if there is a Union involved it is going to get kicked out…

The issue is that Python was altered in a way that is going to break a plethora of software that uses the Windows API and specifically COM objects… The issue was placed 4 years ago… and then poof… out of the blue they fix something that honestly was never really broken in the first place.

How many people here can say they have seen this bug first hand??? probably 0. if it was such an important change to make that caused a complete disregard for killing software then how come there is not a whole lot of people complaining about this problem.??? I am almost wondering if this is a NIX related issue, the OP of the bug report was using a NIX based OS when the report was made. and I also think that other people that were involved in that bug report were also running a NIX based OS…

Now I am not denying that there may be a problem… But the bad impact of fixing it far exceeds the benefit of fixing it. it is not a bug that produced a large number of complaints. so it is not really effecting anything. It is so important a bug to fix and to allow it to break all kinds of code that the report sat for 4 years before anything was done… and then NO notification that this change was going to cause such a massive impact especially on Windows and using COM objects.

I have never had any crashes due to comtypes it’s self for no apparent reason… and you wouldn’y even know if comtypes caused this crash… Because it is an application crash… there is no trace back. you would have to be running a debugging version of python and have a crash happen so it could be debugged and to see where the iossue is… but that is only going to get to so far… the issue could still b from how comtypes is being used… the debugger for the c code is not going to give you a walk through of the application code…

However if you compiled both comtypes and the application code using cython you can then debug the whole process…

I just stumbled upon a bug in ctypes dealing with structures and passing them through a function call. for some reason the damned thing would want to get GC’s but something got screwed upo with the reference count… solution… move the code so i didn’t have to pass the thing… This is an issue within python code… I do not know as to why this was happening… but python was trying to gc something it shouldn’t have been. some how some way with something that i was doing was causing it to get flagged for a GC in a local namespace of a function when there was actually still a reference to it in another function…

I thought that in python a union and a structure are the same thing…

In the python bug report they mentioned both structures and unions but the majority of the conversation was geared at unions and there seems to be only an exception being thrown if a union is being passed. which is strange because I had thought that in python a structure and a union are essentially the same thing. But if it is only throwing exceptions for passing a union then the code below should work. The other thing that is not right is that they provide no solution of existing code. they made the change and left people having to scramble and search out a solution… Kinda messed up actually.

import ctypes

class SomeStruct(ctypes.Structure):
    class _DUMMYUNION(ctypes.Union):
        _fields_ = [
            ('SomeField', ctypes.c_ulong)
        ]    

    _anonymous_ = ('DUMMYUNIONNAME',)
    _fields_ = [
        ('DUMMYUNIONNAME', _DUMMYUNIONNAME)
    ]

Some initial learning of libffi code made me thinking this is too big feature (in libffi) which probably requires implementations for all possible ABI’s (architectures). Also it will take too much time (at least for my level of expertise).

I’m going to start from simple example provided by @tilsche in https://github.com/libffi/libffi/issues/33#issuecomment-204694387. Also I will try another workaround based on .from_address() method of the Structure which also allows re-interpreting memory content without direct usage of a union (this is what I have experience with).

After trying all workarounds on this example, I can proceed with comtypes.

This link tells that the problem is not in undefined behavior: https://stackoverflow.com/q/11373203/3648361

So the reason of the prohibited functionality is that this libffi issue is still open: https://github.com/libffi/libffi/issues/33

Alright, I’ve fixed this by reverting to 3.7.4.