godot: Window per-pixel transparency does not work on hybrid GPU systems (NVIDIA Optimus)

Thanks to one of our users I think we figured out the issue with Vulkan and now also OpenGL transparency on newer (Nvidia) drivers. The issue seems to be caused by a new driver feature that forces OpenGL and Vulkan applications to render onto a virtual display backed by a DXGI swapchain which is supposed to enable additional features and compatibility with VK_KHR_display. It seems that this was also introduced in AMD shortly after due to user feedback.

It can be turned off by switching Vulkan/OpenGL present method in the nvidia control panel to Prefer native. I am not yet sure how the option is called on AMD. From a quick look at VK_KHR_display it might be possible to support transparency without forcing the user to change this setting by using vkCreateDisplayPlaneSurfaceKHR instead of vkCreateWin32SurfaceKHR but my knowledge of both Vulkan and OpenGL is not quite high enough so this would need to be investigated by someone more knowledgable.

shows a gray(white?) color shortly and then become transparent

Resizing windows is relatively slow (especially when using Vulkan), as render buffers need to be reallocated every time the window is resize. Also, empty parts of windows are white on Windows. I’m not sure if there’s a way to override the color that is used for empty parts of windows (you’d need black in your case).

Well that might be the case for you. But it is not working for everyone. I have tested this myself with both a 1070 and a 4090. Vulkan is not working for any driver starting from 537.58 all the way up to 549.29. I have reports from various users with basically any GPU from a 660 up to 4090 that have transparency issues on Vulkan. I also have reports from various users with AMD GPUs. On hybrid Systems it generally works if godot is running on the iGPU or if an external Monitor is connected for some reason.

OpenGL seems to work better in general but also not for all users. So no, this issue has not been fixed yet.

I am currently giving users the option to choose at startup between Vulkan, OpenGL and a hacky DirectX interop mode as well as switching between GPUs. Generally it seems that there is some working combination out of those for everyone. Its just not the same solution that works for everyone.

Yes latest drivers. Vulkan is not working since around September, both AMD and Nvidia pushed driver updates that broke it. Nvidia has claimed twice with recent driver updates to have fixed the issue but they didnt. OpenGL is working on most systems. On some systems you have to force the iGPU via --gpu-index. Downgrading drivers to versions before September is also an option but not a good one.

There seems to be some issue with the latest nvidia and amd drivers. Prior to the september driver updates transparency with vulkan worked fine. Now only OpenGL is working on my dedicated GPU. However, if I switch to my iGPU Vulkan transparency is working.

I noticed that on my iGPU it does not matter if I set

get_tree().get_root().set_transparent_background(true)

to true or false. As long as I have per pixel transparency enabled and

DisplayServer.window_set_flag(DisplayServer.WINDOW_FLAG_TRANSPARENT, true)

set to true the window is transparent.

I inspected the frames with Frame Doctor and the swapchain image always has a transparent background irrespective of set_transparent_background. For some reason the background color changes between black and grey though. Also reverting WINDOW_FLAG_TRANSPARENT to false does not make the background opaque again.

The only difference I could find so far is that my nvidia card only supports VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR while the iGPU also supports VK_COMPOSITE_ALPHA_INHERIT_BIT_KHR however this is true with both the latest driver and the nvidia 531 driver where transparency still works.

I did some further testing compiling the engine myself. Forcing https://github.com/godotengine/godot/blob/066e7d483a8f699fd94e7133ac23978cd460a72d/drivers/vulkan/vulkan_context.cpp#L2128 to VK_COMPOSITE_ALPHA_OPAQUE_BIT_KHR or VK_COMPOSITE_ALPHA_INHERIT_BIT_KHR has no change on transparency on both the iGPU and my dedicated GPU. So this doesn’t seem to be the culprit.

Also while I can only vouch for the tests I did myself. I have reports from over 100 users that observed the same issue. Reverting drivers to pre september or switching to OpenGL did resolve this (for most of them at least).

Steam and other overlays works as a Vulkan layer (or by using DLL injection for OpenGL), by intercepting and modifying host app draw calls, it has nothing to do with window transparency support.

It’s possible to do transparency by rendering to the offscreen buffer and updating the layered window manually, but it is extremely slow (good enough only for a static UI).

so maybe it’s just a matter of updating that function to also disable DXGI Swapchain.

Note that this should not be forcibly disabled, as many users like to enable it to benefit from better framepacing and lower input lag. In my experience, using a DXGI swapchain also tends to make RTX HDR play better with Godot when using a Vulkan or OpenGL renderer.

It’s OK for the profile to default to Prefer native, but it shouldn’t force the option if the user chooses something else.

There’s also a proposal about using DXGI out of the box for the Vulkan renderer, in which case this feature wouldn’t work anymore unless we add a project setting to toggle it: https://github.com/godotengine/godot-proposals/issues/5692

It’s not. Still having that issue on both AMD and Nvidia

So I actually found an application that renders a transparent window with 3d data that works on laptops.

https://github.com/BoyC/GW2TacO

Its DX11 but maybe someone with more knowledge about rendering can find out what the magic ingredient is. The only difference I found is that it is indeed using DwmExtendFrameIntoClientArea instead of DwmEnableBlurBehindWindow. Dont understand enough to know if that is the reason though

OpenGL (Compatibility) renderer seems to work much better with transparency and should work on all GPUs.

While opengl does indeed fix it on some laptops, it doesn’t work on all and for some vulkan works actually better than opengl.

And in case of Windows, you can also cut the window to the specific shape using DisplayServer.window_set_mouse_passthrough (on macOS/Windows it will only affect mouse events, not a window look).

the cutout stuff is actually quite limited. It’s really only useful to change the border of the window. Or have a single hole in the middle