godot-cpp: Internal: stack overwrite on _call_native_mb_ret with godot::Error return value

Godot version

custom

System information

Windows 10 Visual Studio 2019

Issue description

for R is godot::Error, a 4-byte enum:

template <class R, class... Args>
R _call_native_mb_ret(const GDNativeMethodBindPtr mb, void *instance, const Args &...args) {
	R ret;
	std::array<const GDNativeTypePtr, sizeof...(Args)> mb_args = { { (const GDNativeTypePtr)args... } };
	internal::gdn_interface->object_method_bind_ptrcall(mb, instance, mb_args.data(), &ret);
	return ret;
}

before object_method_bind_ptrcall:

image

after object_method_bind_ptrcall:

image

It has written 8 bytes and triggered stack guard detection in Visual Studio. I believe this is real and it may be caused by VARIANT_ENUM_CAST assuming all enum are int64_t and thus overwriting the 4-byte stack variable on encode. reduz would probably know instantly if this is a real issue.

If it becomes relevant, I can try to reproduce this with an official build, but it is hard to do right now because of problems in GDExtension that require me to run various patches. In any case this would be confirmed and fixed by inspection if it is previously unknown.

Steps to reproduce

Debugging extensions code, and by inspection.

Minimal reproduction project

No response

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 1
  • Comments: 20 (10 by maintainers)

Most upvoted comments

Even if its not pretty, but this should already be sufficent.

template <class TEnum>
using int_rep_t = std::conditional_t<
    sizeof(TEnum) == 1,
    std::uint8_t,
    std::conditional_t<
        sizeof(TEnum) == 2,
        std::uint16_t,
        std::conditional_t<
            sizeof(TEnum) == 4,
            std::uint32_t,
            std::uint64_t
            >
        >
    >;

Ok I might steal your code then if I get motivated. I have the advantage of being set up to test already. Unless @RiederAlex wants to do it?