CsWin32: COM interfaces returning structs use wrong calling convention

For people (like me) that are searching for a performant workaround for the issue, you can disable COM mashaling and manually use new MemberFunction calling convention which resolves the issue:

private unsafe D3D12_CPU_DESCRIPTOR_HANDLE GetDescriptorHandle(ID3D12DescriptorHeap* heap)
{
    var vtable = *(nint**)heap;
    var getDescriptor = (delegate* unmanaged[Stdcall, MemberFunction]<ID3D12DescriptorHeap*, D3D12_CPU_DESCRIPTOR_HANDLE>)(vtable[9]); // index in vtable needs to be found by inspecting code generated by cswin32
    var getDescriptorResult = getDescriptor(heap);
    return getDescriptorResult;
}

@jkoritzinsky, it’s worth noting that this is compatible but also not quite right either:

change the method to return void

The underlying ABI also returns a pointer to the return buffer parameter and so for a method such as:

virtual D3D12_RESOURCE_ALLOCATION_INFO STDMETHODCALLTYPE GetResourceAllocationInfo( 
    _In_  UINT visibleMask,
    _In_  UINT numResourceDescs,
    _In_reads_(numResourceDescs)  const D3D12_RESOURCE_DESC *pResourceDescs) = 0;

The fixup on C# should/can be:

public D3D12_RESOURCE_ALLOCATION_INFO GetResourceAllocationInfo(uint visibleMask, uint numResourceDescs, D3D12_RESOURCE_DESC* pResourceDescs)
{
    D3D12_RESOURCE_ALLOCATION_INFO result;
    return *((delegate* unmanaged<ID3D12Device*, D3D12_RESOURCE_ALLOCATION_INFO*, uint, uint, D3D12_RESOURCE_DESC*, D3D12_RESOURCE_ALLOCATION_INFO*>)(lpVtbl[25]))((ID3D12Device*)Unsafe.AsPointer(ref this), &result, visibleMask, numResourceDescs, pResourceDescs);
}

This is also slightly more convenient than having an additional return statement that is effectively return result. You can see an example of this behavior here: https://godbolt.org/z/4EofM4 (and can also go browse the MSVC source if needed).

In this case, CsWin32 will need to manually adjust the signature to insert the return buffer since .NET will not do the correct thing without the new CallConvMemberFunction type.

Ok, so I need to modify the return type to be a pointer, and I need to add that same pointer type as an additional parameter in the method as well (always in last position).

You’ll need to add the additional parameter after the this parameter and before all other parameters.

And I do this when the return type is a struct that wasn’t a typedef on the native side. For COM interface methods only. Right?

Yes.

.NET accidentally gets the behavior right, since the native typedef HRESULT is treated the same as the underlying int type. This accidental success is the “back-compat” case mentioned previously.

do we know if the accidental success here is RyuJIT/Legacy JIT only or if it also applies to Mono and Mono/AOT?

I do not know if Mono has the same problem with their COM Interop support, but I wouldn’t be shocked if it did. I haven’t had a chance to test it yet. They’ll definitely have the same problem with function pointers though.

All our COM structs define vtbl’s with function pointers as fields, and they work on net472.

Do you have an example of what is being generated? I recall fairly simple examples showing this fails.

not sure about what the ABI is for large primitives, never seen a COM method return a primitive larger than 8 bytes

D3D12 has a few, one of which is the GetResourceAllocationInfo example I keep giving. D3D12_RESOURCE_ALLOCATION_INFO is a 16-byte struct containing 2x UINT64 fields. You can find many examples (currently 107) of these types of fixups by grepping for result; in my TerraFX.Interop.Windows project: https://github.com/terrafx/terrafx.interop.windows/tree/main/sources/Interop/Windows, https://source.terrafx.dev/#TerraFX.Interop.Windows/um/d3d12/ID3D12Device.cs,8b9b043be241c1ea. I’ve provided my own grep results here for convenience: result.txt

Edit: I misread the comment as struct, not primitive. As far as the ABI is concerned, things like __int128 aren’t true primitives but rather their own type. __m128 and friends are primitives and respected by the ABI. These correspond to Vector128<T> and friends in .NET and are only available in .NET Core 3.0+. They are currently blocked in P/Invokes because we don’t correctly handle the return scenario on Windows.

The correct solution here would be to do a two-pronged approach:

For the typedef structs, you can either do one of the two following options:

• Use the underlying type as the return type for the function pointer. This will enable the correct results if the generated code ever uses CallConvMemberFunction.
• Use the typedef type as the return type. This ends up using the “non-member function” calling convention, which will work in this case. This will continue to work if the code-gen uses the built-in COM support as well (this is the legacy behavior Aaron mentioned), but not if the code ever uses CallConvMemberFunction.

For the non-typedef structs, you can do one of the two following options:

• Manually transform the function pointer/COM method signature to have an out return buffer parameter after the native this parameter and change the method to return void. This will be necessary in the “pointer-free” mode you mentioned. The built-in COM system will never support using the natural signature for this case because it breaks the other case pretty severely.
• Use CallConvMemberFunction.

We’re introducing CallConvMemberFunction to enable using the natural function signature with member functions that return structs since we can’t change the built-in behavior without breaking people that rely on the “feature” that their HRESULT struct type can be used in place of an HRESULT.