runtime: CLR JIT does not use correct ARM32 calling convention for 8-byte return structs

This issue affects UWP, so I’m not sure that this is the correct place to file this or not but:

Reference to stack overflow question

After nearly a week of research I’m ready to say that the CLR is causing issues on ARM devices but I’m not quite sure why. To summarize:

• I have a C function that returns a struct with two word-sized types (so a total of 8 bytes on ARM32) and takes two parameters (both data pointers)
• I have a P/Invoke call for this function from C# in which all the types are modeled after their C counterparts
• The Procedure Call Standard for the ARM Architecture states this about return conventions: “A Composite Type larger than 4 bytes, or whose size cannot be determined statically by both caller and callee, is stored in memory at an address passed as an extra argument when the function was called (§5.5, rule A.4). The memory to be used for the result may be modified at any point during the function call.” I take that to mean that it will be the first argument, to be passed into r0
• However, the CLR passes the first parameter into r0 instead of a pointer to where the result should be stored and the second parameter in r1, causing invalid memory accesses.
• .NET Native does not produce the same issue, suggesting it is limited to JIT

If I “lie” and say [StructLayout(Layout.Sequential, Size = 12)] on the return value, the problem goes away. If I add a dummy int32_t onto the end of the struct on both the managed and unmanaged side then the problem goes away. This suggests to me that the problem only affects return values of 8 bytes in size. Perhaps the CLR is mistakenly applying the rules for fundamental 64-bit types instead? That would be:

“A double-word sized Fundamental Data Type (e.g., long long, double and 64-bit containerized vectors) is returned in r0 and r1.”

I extracted just the bad parts into a project that should reproduce the issue. The behavior here is undefined so it might return, but the value will be garbage. Pay attention to the change in value between the managed side and the native side of FLEncoder_Finish. To make this as easy as possible, I precompiled the native ARM DLL, but if you need the source it is here (The PDB file should have symbol info in it). Running without .NET Native causes the issue, running with .NET Native does not.

This issue has been reproduced on my Raspberry Pi on Windows IoT, but was originally reported from a user deploying on Windows Phone.

Spike.zip