godot: GDExtension library cannot be reloaded while editor is running

Now that we have https://github.com/godot-rust/gdextension/ I was able to play around a bit with the new system and, I don’t want to sound too harsh, but GDExtension feels like a substantial usability downgrade for me as a game developer due to this issue. Let me elaborate:

The way I develop games with Godot 3.x using a native language (Rust in my case, but this applies just as well to C++ devs) is pretty much the same as one would do in regular GDScript: I write script classes, only that instead of using gd files, I use gdns files. This is and incredibly productive setup, because I get the full editor experience, but all my game logic is written in a language that makes refactoring and scaling my game much easier. Note that my use case for a native language has nothing to do with developing editor extensions or speeding up critical pieces of game logic: My game logic is entirely written in a native language so I can benefit from the mature static analysis tooling around it.

Now, I know I’m not the only one using Godot like this, but I can only speak for myself. My issue with the transition to GDNative to GDExtension is that it takes away value from me as a user without really giving me much in return. Let me explain with an example:

In Godot 3.x, this is how I work when I want to create some new behavior:

• I start by creating a new rust file, writing a new struct (a godot class), and exposing some of those struct methods to godot. It looks like this. That link is a Zeppelin character controller. Does that read lightweight and simple like a version of GDSscript with curly braces and static typing? Well, that’s how it feels to me 🙂.
• Next, I jump back to the editor, create a new node (in this case, a Spatial), and use the asset finder to locate the Zeppelin.gdns file. I drag and drop this file into the node, and the script is set automatically. Yup, I didn’t have to do anything else, it’s automated: The gdns file was generated by a build script and my dll was recompiled when saving the file (Rust has cargo watch, but C++ devs can use entr to recompile their code on file save).
• At that point, I can run the scene and start iterating on the logic. That process will typically consist of launching and closing the game multiple times, making small tweaks to the code. During this process, I usually add properties to my class (The equivalent of export vars in GDScript), and tweak those in the editor. It is very important for me to be able to add new properties on the fly, because it’s impossible to predict which parameters I will want to expose beforehand.

This all works wonderfully and makes me feel as productive as it gets! But now, let’s take a look at my pain points in Godot 4 via GDExtension:

• Creating a new class requires an editor restart. This is true even if the Windows DLL locking issue is solved. I’m not even on Windows. Unless GDExtension gets full hot-reloading like GDNative used to have, every time I create a new script I will have to restart the editor, loosing my state (all the open tabs, prompting to save unsaved changes I may not really want to save, …). This breaks my flow and forces me to retreat to other styles of development that simply ignore the editor and use Godot as a rendering library, which IMO takes away most of its value.
• But even worse: Adding or removing properties also requires an editor restart. This makes exporting vars from my native scripts far less useful, when it used to be one of the biggest productivity boosters for me in 3.x.

I really don’t mean any of this in a bad way. My hope is that, by explaining my workflow in detail, I will help developers understand why so many people seem to care about this issue (I mean, look at the upvotes!). There is a niche (yes, we’re not that many, but we exist) of developers that has seen a considerable degradation in their workflow when transitioning from Godot 3 -> 4.

Basically, I just came to suggest that, even if the DLL locking bug is solved, what lies underneath is not the productive experience developers used to have in 3.x. I understand that might not be a priority, and I know I’m hardly Godot’s target user by any means, but I’d like to raise awareness about why this is an issue and how it breaks developer’s workflows exactly.

FYI, PR https://github.com/godotengine/godot/pull/80284 (which attempts to implement hot reloading for GDExtension) is ready for testing.

It doesn’t do everything described in this issue, and there may still be a use case for something like cppscript even if/when GDExtension supports hot reloading, but it’s a start! If we can get the PR tested, reviewed and merged in the next ~2-3 weeks, then it’ll be included in Godot 4.2. (If not, that just means it’ll have to wait until Godot 4.3. 😃)

My suggestion would be to avoid error-prone cases by (with user confirmation) closing scenes and editor plugins during the reload and reopening all scenes/plugins after the GDExtension is reloaded, since it’s not that common.

I wouldn’t be super happy about this tradeoff, because it would still feel worse than the current GDNative experience.

One important thing to consider here is the editor’s state. There’s lots of UI state that Godot doesn’t really serialize, and these would be lost every time you reload your extension. Moreover, “saving” and reopening a scene is a destructive operation (did you really intend to save your changes?). A few things that would be annoying about this workflow. Maybe some of these points can be fixed, but I figured it’s good to bring them up:

• The undo stack would be nuked every time you make changes to your native code.
• The viewport camera angle and its settings would reset.
• In a native-heavy project, every open scene would need to be reloaded. This means reloading every open scene every time you make a tiny change to your code.
• Some transient state, like text boxes that are partially being filled but not yet serialized would vanish.

These are not big things on their own, but together they add up and become a considerably worse UX when compared to using GDScript for gameplay logic.

Again, I hate to be that guy, but I have to remind you all that all this worked on Godot 3 just fine 😅. It’s not like people is asking for something impossible. At the very least, we should consider solutions that aim to bring something close in spirit to the old workflow back.

I’m also a bit concerned about the “duct tape” effect here. If we prioritize a quick (duct tape) fix to make reloading on Windows work, I suspect the incentive to polish the experience any further on top of it would drop considerably: “What’s the harm in a little editor restart every time you register a new class?”, “When you hit play, the new code is loaded, after all”.

Please note that I’m not trying to miscaracterize anyone with the above comment. I know everyone in this thread understands the underlying issue 😃. But this is a genuine concern, because it’s often easy to loose track of your long term design goals and UX needs once an 80% solution lands and many of the people who were asking for a better design just learn to live with it in the spirit of getting work done, then start working on their own (worse, but functional) workarounds.

The way I develop games with Godot 3.x using a native language (Rust in my case, but this applies just as well to C++ devs) is pretty much the same as one would do in regular GDScript: I write script classes, only that instead of using gd files, I use gdns files. This is and incredibly productive setup, because I get the full editor experience, but all my game logic is written in a language that makes refactoring and scaling my game much easier. Note that my use case for a native language has nothing to do with developing editor extensions or speeding up critical pieces of game logic: My game logic is entirely written in a native language so I can benefit from the mature static analysis tooling around it.

Just want to echo and really emphasize this comment, as I think this is the main use case for extending the engine for use with other languages, which is primarily at the scripting layer not necessarily core engine layer.

Very happy to see this addressed! Great work everyone 🚀

May i suggest interim solution while hot-reloading is not supported?

• I agree that the .dll should be copied into private location, so the file is not locked.
• Meanwhile Godot watches it for changes, and if it is it prints a warning like: We detected changes to <file>. Unfortunately, currently Godot does not support hot-reloading GDExtension yet. Please reload the project to update.
• The user then can have time to prepare stuff, like saving or even updating other DLLs.
• When user hit reload project, it releases the lock, copies the changes, then reloads the project.

I just started learning godot, and pairing it with godot-rust for similar reasons that others have mentioned - to get a more pleasant coding experience. This issue really started to bug me, landed here and I am so pleased to learn it’s already fixed in 4.2, and implemented in godot-rust also. All praise open source

I join the problem. It is impossible to work without auto-reload of compiled code. Also GODOT is great engine and i really love it, but this is serious issue

Having released one non-trivially sized game with this in 3.x and been looking forward to 4.x for a while, I don’t want to be overly dramatic, but this issue is enough of a reason for me to not consider using Godot 4.x for any future projects.

In my opinion this is not being dramatic, this is a valid point and I’m at the same boat. Having just prototyped an idea with Unity I have been eyeing Godot 4.x as the main platform. This unfortunately might be big enough of an issue for me to pass Godot for that idea and wait if this side of the engine matures better with time.

Like @setzer22 wrote well, having a mature static analysis tooling makes all the difference when writing game logic/systems. GDScript (without going into detail here) has too many flaws for me to enjoy writing large parts of the game in - smaller bits are fine though. And on top of that the prototype in question has liquid dynamics associated with the gameplay and GDScript performance is not suited for such a thing.

Might this be of interest? https://github.com/fungos/cr/ 🤔

Hmm, quite an interesting library. Maybe writing an engine module for Godot that acts as a host for a plugin might work but that could be quite hacky, require lots of macrofoo (for proxying) and maybe - as this is quite large of an issue - this would be better to fix at engine level rather than bubblegum fix 😅

Just want to echo and really emphasize this comment, as I think this is the main use case for extending the engine for use with other languages, which is primarily at the scripting layer not necessarily core engine layer.

Personally I’m not even sure if I ran into someone who tried to extend the core engine this way. Maybe I’m biased since I personally also used it for “scripting” (or specifically replacing GDScript with Rust to get both performance and more developer friendly environment), but it does seem like this is the actual use case?

Having released one non-trivially sized game with this in 3.x and been looking forward to 4.x for a while, I don’t want to be overly dramatic, but this issue is enough of a reason for me to not consider using Godot 4.x for any future projects. At least for me the main reason to consider Godot is getting an editor that’s integrated into the dev workflow. One already has to make a sacrifice compared to Unity and UE4 where inspecting the scene during play mode is much more interactive, while in Godot it’s much more detached. But this is a sacrifice that is worth making for some other benefits, namely the nice integration with a native language 3.x provided.

But if this relative interactivity with the editor is taken away with 4.x, it creates another point of friction between development and using the editor., making the editor even less useful for people who use native scripts. Maybe some can develop without using the editor and only using Godot as a renderer, but at that point I’d question the benefits of using Godot at all, as many viable alternatives exist. To me the main benefit of Godot and GDNative was that I could get both simple scripting, as well as tight integration with Rust. It wasn’t perfect, but it worked well enough where after the initial setup I could just build my game without thinking about it too much.

Being simply a consumer of the engine and being 100% focused on making games I don’t see inside the engine and don’t understand the intricacies or needs for doing a big rewrite to GDExtension. But from purely a game developers perspective, I really don’t understand why I’d even want GDExtension, and would 100% be fine with having “4.x features” with the same API as GDNative in 3.x. I understand there are probably internal reasons, and that this is really not what people want to hear after having spent months working on it, but for me as a simple consumer who had many issues with Godot 3.x, GDNative was really not one of them. If there was an option to get the nice things in 4.x (better 2d sprite batching, better tilemaps, etc.) while having the same GDNative extension, I’d take it over any extra power from GDExtension, simply because as a small game developer I don’t imagine ever needing to extend the engine over just building things as a feature. Maybe this enables certain class of more ambitious games that were harder to make with engine mods previously, though if that’s the case I’d have to question if this is actually Godot’s target audience.

I can’t believe this has been solved, I was expecting this to take a couple of years, it’s really a pleasant surprise

* writing my game with GDExtension.

To me, this is the main use-case for developers no? At least i would imagine most developers being interested in implementing game logic in language of choice, as opposed to customizing engine internals and things that behave like native modules in language of choice.

You then have the issue (like in GDNative) that you cannot attach scripts to GDNative objects (because it already has a script, the C++ script).

For us, our typical pattern for this is to add ‘components’ to nodes as a child node, which pretty much enables you to attach any script or behaviour to any node. This ends up being really flexible because you can do components that are composable, and more flexible than the typical ‘implement all customization in a OO subclass’ approach. Ex, if I need a custom script for a node, instead of strictly using OO and making a subclass for some modified behavoir, I can attach a component. Ex:

So this is actually relatively easy to do, just add a basic child node and attach a custom script (native or not). It doesn’t matter if the root node is a GDNative object or built-in engine object or not, the behaviour or scripting of the object is still extensible. Here we add 3 behaviours to a node, and each behaviour can act on it’s parent get_parent() and potentially make whatever assumptions about it’s parent as necessary.

So for cannot attach scripts to GDNative objects you can typically add a child node, attach script as normal, and reference the parent component via get_parent() make assumption about what ‘type’ parent node is, and continue on.

As far as I can tell, and correct me if wrong, but one of the main motivations behind the current GDExtension design is the ultimate ability for extension developers to distribute compiled/native plugins/libs in a unity-like assets store / extensions store. (Which can then be ‘plug-and-play’ with the pre-built godot binares and export templates). And I think that’s a great goal to strive for. But I think there is a huge, (if not the majority) of interest is in implementing game logic in a language other than gdscript. And I believe for achieving that goal, workflow and usability are absolutely 100% the most important part of that equation.

we should indeed implement a C++ ScriptLanguageExtension and bring back the “act as script” behavior.

I think this could perhaps be really important, if there is no usable workflow for developers… The current proposed workarounds of reloading all scenes, reserializing everything, losing undo-redo stacks, losing any in-memory editor scene state, etc, are pretty much showstoppers for this use-case.

Do you mean the infrastructure to support custom scripting languages (what you’re referring to as ScriptLanguageExtension?) is already there?

Yes, my understanding is that this is indeed the case, and the setup I mentioned above (a module-like gdextension which exposes a reloadable scripting language) can be achieved by appropriately implementing ScriptLanguageExtension, ScriptExtension and the GDExtensionScriptInstanceInfo interface.

Note that I do not know of any actual implementation of those extension classes, but they were designed with that scope in mind (cfr https://github.com/godotengine/godot/commit/360dea5348652d8806d33598e111651afb3d193a)

I tried to create a plugin for Godot that uses NativeExtensionManager class in GDScript to manually control loaded extensions.

You can unload the extension, compile new dll, replace the old one and then load the extension again. Now this does really work but it breaks the engine.

It either

• crashes after hitting play with new dll in place.
• breaks gdscript somehow (my lambdas that worked fine before just break with obscure errors, memory corrupted maybe?).
• outputs ‘“scene/resources/packed_scene.cpp:178 - Node ‘./TestNode2D’ was modified from inside an instance, but it has vanished.”’ and removes the whole extension node from saved scene (ouch, this should not happen right?!).

After these I just gave up. Unfortunately I need native extensions due to performance (and somewhat disliking GDScript) and restarting engine each time is annoying workflow for rapid development and IMO somewhat undermines the idea of dynamic GDExtensions.

I understand “hot reload” is hard to get right (UE as an example) and I don’t think it even needs to be pursued but replacing libraries between running the game from editor should be doable without the “hot” magic?

edit: typos

This is certainly a big issue for me as well, but I’ve noticed something after reading all of this. One of the major points of issue for the engine developers seems to be that GDExtensions can register new types that would break the engine if they were removed at runtime. But as @Faless and @setzer22 talked about, that’s not a requirement.

My suggestion is to have a set of registration operations that are considered “reload-safe”. This would include registering new scripts and new resource types, since both are serializable. Any classes registered that are more complex would mark the extension as “reload-unsafe”. The extension could include in its metadata if it’s trying to be “reload-safe” and an error could pop up if that contract is violated.

Implementation-wise any non “reload-safe” ClassDB registrations set a boolean flag for the GDExtension that called them, and when loading is done the editor checks that flag against the extension’s expected “safe-ness”.

Combine that with the above recommendations of copying the dll/so/dylib into the .godot folder with a unique hash and, it seems to me, that you have the best of both worlds.

TL;DR: GDExtensions that ONLY register script classes and fully-serializable resources should be considered “hot-reload safe”. Anything else should require an editor restart.

@TheMasterofBlubb I was able to get godot to the point that it detects the gdextension library change after a recompile. It then automatically recommends a restart. You can check out my fork linked above.

@MachineMakesNoise sounds like you have some experience with this. I made an attempt and it doesn’t crash on macos. The library loads, but at some point I believe the godot::Object, _extension and _extension_instance pointers need to be updated. I accomplished this by only swapping out the library pointer of the GDExtension object.

For godot-cpp I think this would involve having the classes/wrapped.hpp GDCLASS macro keep track of it’s GDExtension and use some kind of signaling mechanism tell all the objects to refresh their internal pointers. There’ll need to be some mechanism for saving/restoring properties.

Compile with my partially working code using CXXFLAGS="-DSUPPORT_DYNAMIC_GDEXTENSION_RELOAD"

Also see this regarding compiling dynamic libraries(at least on macos) godotengine/godot-docs#7284

The main difference is that GDNative worked that way via the Script interface, i.e. C++ objects were just scripts attached to nodes.

This is not how GDExtension work, where instead you register functions and classes in the engine like internal code do, and they behave like modules (and like modules can’t be easily unloaded/reloaded).

Indeed 👍 That much is clear. GDExtension was created in an attempt to make engine extensions easier and to have a way to phase out the old “custom modules”. But it is not working as a GDNative replacement.

You then have the issue (like in GDNative) that you cannot attach scripts to GDNative objects (because it already has a script, the C++ script).

What you’re presenting here as a bad thing is the main feature I, and other people on this thread, are asking for. We want to use native languages for scripting, that’s the idea! Of course I won’t be able to attach a script to a node that already has a script, it’s not an issue, that’s how Godot works.

It seems to me most of the friction is caused by the existence of 2 different use cases.

Agreed! And the two use cases seem to be different enough that they need different solutions. GDExtension does not feel like a GDNative replacement in that regard.

I’m not sure how the 2 cases can be reconciled, maybe for the second one we should indeed implement a C++ ScriptLanguageExtension and bring back the “act as script” behavior.

Then you would load the libcppscripts.dll (changing this require restarts), which would then load libmygame.dll where your C++ scripts are (which is much easier to reload as a script).

This sounds nice and reasonable. The kind of thing that would make me say Godot truly supports “any language” for scripting. I’m only speaking for myself here, but I’d be perfectly happy with a solution like this that lets me write game logic in my language of choice, and I would understand extensions being more integrated to the engine and thus requiring editor restarts.

But I’m a bit confused by the rest of your message(s). Why are you talking specifically about a C++ implementation? Do you mean the infrastructure to support custom scripting languages (what you’re referring to as ScriptLanguageExtension?) is already there?

works nicely! ty!

I feel like from the moment gdextension was planned as a replacement to gdnative, it had to account for reload. Because at this point even if there are some improvements in features. It is a HUGE downgrade in actual usability. Like because of it, i can’t even consider full c++ games on godot 4, reopening the editor all the time is a pain, and on godot 3 i had made a class factory anyway that autogenerated the gdns files so the workflow was waaaay faster.

I feel like while gdextension didn’t have feature parity with gdnative, gdnative should not have been deprecated and should have been another option alongside gdextension

Le jeu. 25 mai 2023 à 11:23, Ignacio Roldán Etcheverry < @.***> a écrit :

I don’t think there’s anything that can be considered safe to reload, not anything useful at least. One of the main problems is inherent to the fact that native extension classes are treated the same as engine classes by ClassDB. And any one can grab a pointer to a method of one of those classes and cache it. That’s what language implementations do, otherwise calls would be too slow.

Fixing this could require changing native extension classes to be treated differently, which kind of defeats the point. Another option could be to make the methods (and other stuff) refcounted, to avoid references to freed memory after a reload. Making a call one of these references should either work or result in a non-fatal error message if a method with that signature is no longer present after reloading. Additionally, ClassDB should notify anyone interested when something is reloaded, to allow them to adjust (their bindings, for example).

Even then, I don’t know if that would work in practice, and it’s only one of the problems.

This is the main reason I only treat native extensions as library dependencies. For code I need to iterate on a lot, there’s only the script system for now. Unfortunately many language bindings (like cpp and rust I think?) that were using the scripting system in the GDNative days, are now using native extensions classes only.

— Reply to this email directly, view it on GitHub https://github.com/godotengine/godot/issues/66231#issuecomment-1562574705, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD4UF4K5B5LQCB5KQW3YYQDXH4QHTANCNFSM6AAAAAAQSNTYVY . You are receiving this because you commented.Message ID: @.***>

To me, this is the main use-case for developers no? At least i would imagine most developers being interested in implementing game logic in language of choice

As far as I understand, to achieve this, the language of choice needs a ScriptLanguageExtension implementation in C (or godot-cpp), and the library needs to handle reloads via the various reload* functions. This should already be possible.

To my knowledge, if we implement a CPPScriptLanguage of some sort, it will likely only be for C++, and each other language (rust/python/etc) would need to implement their own ScriptLanguageExtension.

Basically, GDExtension let you extend the engine, this includes adding a scripting language. If you implement a scripting language you can handle reloads, if you implement a module, you can’t.

The godot-cpp bindings currently implements a C++ wrapper for module-like extensions, adding a CPPScriptLanguage seems to me the correct solution for developers who want to write C++ scripts, though it’s probably not something that can be added overnight.

With this solution the editor won’t pick up added classes, properties or methods, that will still require a restart, but it will greatly reduce the need for restarting the editor in many situations

This comment @BastiaanOlij is a big concerning, and for me kinda misses a major goal of extension system’s potential. I’m curious what the technical challenges/issues are compared to gdnative in 3.x, because this is currently doable in 3.X correct?

Those 3 things (adding/modifying classes, properties, and methods) are pretty much the main use case for gdextention for a lot of developers no? The workflow goal would be to:

1. write native code/scripts (in whatever lang).
2. trigger native compilation
3. reload the dynlib into editor’s process space, with a variety of hooks to support implementation of edge case handling.

Ignoring and putting all other native problems aside (that can potentially totally break everything, like say storing memory addresses within your dynamic library across reloads, statics, symbol names, etc), ClassDB in and of itself should be able to support modification all of registered classes, methods, and properties at runtime no?

Back to the good old times when it took quite an effort (and 10+ floppy swaps) to compile my Amiga games 😃

I would love to see this hot swap feature.

Perhaps, have a toggle button which when toggled off would unload all libraries (release locks), let the user recompile, then user could toggle it back on.

Editor could automagically re-lock when it detects file changes after the library was built.

Of course, it would be best to find solution that doesn’t require any button push 😃

You guys are awesome, ty for Godot!

I just encountered this issue. that’s a big issue because it makes godot even LESS viable for c++ games when godot 4 was supposed to be a game changer for perf intensive 3d games

to me it makes no sense

Isn’t that already implemented to some extent by #60597?

One idea I had, from the contributors chat:

Since deleting extension instances is not an option (as it would tear apart scenes), is it maybe possible to replace them with “ghosts” as long as they’re unloaded?

Let’s say you have such a scene:

Node
+-- MySuperCamera (inherits Camera3D)
    +-- Node3D
    +-- Light3D

Then, after unloading the extension, the UI would display it like this:

Node
+-- Ghost (inherits Camera3D)
    +-- Node3D
    +-- Light3D

Internally, Godot would still memorize the type of ghost, and could reassign “MySuperCamera” once reloaded.

If access to a ghost happened while unloaded, there could be multiple ways to deal with it:

• error
• fallback to base class (Camera3D)
• do nothing / dummy impl

Answer from BastiaanOlij:

i was thinking about that too, in theory in the background we could keep the ‘godot class’ that the gdnative class is based on alive and only destroy the extension side, and than recreate on load. It’s not true inheritance we do here, the gdextention instance is separate.

But whether the internal structure supports that, i’m not sure, i still don’t have a complete picture in my head of this.

I find none of the extra features in GDExtension actually useful to my work (I don’t mean that in a bad way: I completely understand the use case, I’m just not an engine developer, I am a game developer). IMHO, I would very much prefer if something like GDNative / Nativescript were brought back so that I could continue developing like before, with each “class” I create being equivalent to a script I can attach to nodes, and not a node type itself.

I don’t think these have to be mutually exclusive! A ‘NativeScript’ (attaching a native compiled script to any generic node) has a really strong appeal as a ‘component’ based approach to game development that can be used in replacement of a GDScript attachment… i.e. really useful for implementing logic in another language, be for whatever other reason. (performance, static analysis, tooling, etc).

GDExtension can be targeted for engine or module developers who are integrating into the engine tighter than at the game logic / scripting level.

I think the later will be a use-case that is utilized to a lesser extent than something like ‘NativeScript’ attachment to any node, so the workflow around a native script attachment use-case imo should be prioritized. This also starts the scratch the surface of ‘inheritance’ vs ‘composition’, as the current behaviour of GDNative in 3.x is perhaps more component based (attaching a script resource to any node), whereas it seems GDExtension is targeted towards an inheritance pattern where the customization of behaviour is more rigidly attached to a defined subclass (which appears to be adding an additional layer of constraints and potential issues).

Perhaps a middle ground can support both approaches?

@BastiaanOlij In my opinion, the main appeal of GDExtension is for extension developers. Say you’re developing a terrain system, or a new low-latency audio server for Godot. GDExtension is clearly better in those scenarios because you can build classes that feel just like regular godot nodes to their users. Everything is integrated, the class appears in the “Add node” menu, and you even get things like autocomplete in GDScript. This is great because more experienced developers can become almost-core engine developers and provide a lot of value to Godot’s target audience. With this use case in mind, it makes sense that there will be a bit more pain involved (i.e. frequent editor restarts): Developing these extensions is a fundamentally different workflow than what regular GDScript users are expected, and the experience has to be optimized for the latter.

But, speaking as someone who has done a lot of GDNative work, I find none of the extra features in GDExtension actually useful to my work (I don’t mean that in a bad way: I completely understand the use case, I’m just not an engine developer, I am a game developer). IMHO, I would very much prefer if something like GDNative / Nativescript were brought back so that I could continue developing like before, with each “class” I create being equivalent to a script I can attach to nodes, and not a node type itself.

I think enabling proper hot reloading only when certain criteria is met would be a reasonable compromise here. Implement reloading, but if Godot detects some DLL registered a new native class after a reload, simply crash (in a safe way, and not letting memory be corrupted). Then we can start thinking about a mechanism to enable iterative game development in native languages, which could be built on top of the current extension system, instead of around it.

That’s my idea anyway 😅 I’d be interested on hearing about other people’s opinions. Would you be happy with a NativeScript-like approach? Or is real native classes, GDExtension-style, something you were looking forward to when in order to build games using native languages (C++, Rust…)?

The hardest part of this seems to be invalidating all of the pointers the library hands to the engine, like init function pointers, GDNativeExtensionClassCreationInfo, Extension Class method pointers, etc. All references to the library need to be dropped before the library can be unloaded.

Edit: Actually my first examples aren’t that hard to deal with, it’s mostly about objects like nodes or custom servers.

Hi, I’m adding myself to the discussion as this is an important issue for me too.

I’m currently evaluating (through prototypes) using Godot with GDExtension so that I have most logic and “model” of my game in C++ while using Godot to implement the “view” (including input handling) of the game. This seems to work as expected so far so I’m almost quite happy - except this hot-reloading issue.

In my case, I need both kinds of reloading, script and class, I think, but it’s not completely clear to me yet how different these are. Also I think it’s ok if I have to explicitly reload for class changes and hot-reload scripts. Would that means we would ideally I have to separate the scripts code into a separate extension?

I also intend to add a Godot Editor GDExtension because I use an unusual build-system (build2) and I would also like to have Godot call it when I focus on the Godot Editor so that if I ever change some C++ code it’s immediately reflected in the editor once hot-reloading works, the same way I can edit GDSCripts outside of Godot Editor and focusing on the editor windows will take the changes into account. Not sure if a GDExtension is actually necessary for that, I was expecting a field in the project/editor for a command to run that must succeed when the editor is focused on, but didnt find it (if it exists already I’m interested).

I have some experience in implementing hot-reloading on Windows (long ago, in C++) but I am not familiar with the codebase of Godot and I guess my time is too limited for diving in this big thing, but if I can help in some limited ways (testing? debugging?) feel free to ask 👍🏽

we’d save and unload all scenes before reloading externals

That was my thought. To build that idea out a little more:

• Add a “reloadable” configuration setting to .gdextension files.
• Extensions without this property act the way they currently do. This could make debugging easier in some cases, and is a minor optimization for mature extensions.
• When loading a “reloadable” extension, make a copy to a private location and load that.
• Watch the original extension file for changes. When a change is detected:
• Save current project to a private location
• Save various metadata to a private location (original project location, list of open tabs, file modified/unmodified states, undo stack, etc)
• Close the project.
• Make a new copy of the extension, overwriting the old copy.
• Open the copied project.
• Reload the saved metadata.

Might this be of interest? https://github.com/fungos/cr/ 🤔

Main takeaway: Godot never directly loads path/to/library.dll, so the lock does not exist.

I think it’s very important for the loaded file to be debuggable in the same way as the compiled one. Not sure if this change might interfere with that.

Hey im fairly new to Godot and started with my first Godot 4 project just recently, but im pretty firm with C++ etc.

For reloading a Library, wouldnt it be an easy thing to distinguish between reloadable and non-reloadable libraries, by simply having 2 mandatory functions for all realodable libraries, which are a serialize and deserialize function.

Sure you would need to make them compatible when making iterations, which creates some extra work for the dev, but overall that would make reloading any custom extension easy and achievable.

When ever a certain extension is realoded:

1. grab all objects that originated from there
2. serialize them
3. save the serialized data next to the object itself
4. load new extension
5. deserialize through new extension library
6. grab new pointers for the objects and put them inplace of the original ones

This would bring some caveats with it, like what to do if the dev breaks his serializer? In that case the editor should shutdown gracefully and tell the dev he did an oops, or reload the old extension (because there the deserializer should 100% work).

A requirement would be to basically wrap all custom nodes into a container that has the extension origin and a data block for serializing.

Just throwing in some thoughs.

Sorry if this is a dumb question as I don’t have that much knowledge how gdextensions are handled inside the engine but isn’t one of the main purposes of InitObject::register_terminator to enable runtime uninitialization of loaded classes?

As in the reload would be something akin to :

1. Lock the engine so it won’t call anything during reregister (don’t know if this kind of lock exists)
2. Call “termintor” to unregister all the extension things
3. Load new dll
4. Call “initializer” to register all the new extension things
5. Unlock the engine again

I’m sure InitObject could be extended to handle old -> new state transfers by introducting callbacks for it if necessary?

I am more worried about the way how Godot handles “orphan” or unexisting classes as it just removes them! Hopefully this can be easily fixed by just showing errors and assigning existing classes to some kind of null/dummy object or similar that does not do anything until the actual implementation is found again. In any case removing the whole object and all things attached to it means lost work…

For me the requirement of restarting to get new properties/methods is not that much of a problem. It is pain yes, but manageable pain 😃 I just want to iterate fast when coding and usually that iteration happens inside few known functions/methods.

I would like to have a Button in the Editor to compile (like C#?) which triggers

1. save all open scenes to a hidden location
2. unload “scripting” gdextension (free shared library)
3. trigger a compilation
4. reload “scripting” gdextension
5. reload all scenes from the hidden location

To work Godot needs some Information

• which gdextension to reload (path should be enough)
• how to compile (terminal command and working directory?)

Maybe a special resouces with the information is enough. The import process would trigger the workflow. Extenal compilations could touch the resource file to automate the process. (somehow avoid recompilation) (This feels more like a hack then a solution)

sava solution sounds similar, so I have no idea if this can work.

Very rough idea, how the new DLL could at least be used for a game’s launch (1st problem):

1. Godot extension config file mentions path/to/library.dll
2. Compiler generates path/to/library.dll
3. Godot keeps a watch on that file, to be notified about changes
4. On a change, Godot copies path/to/library.dll to .godot/libs/library-8172389312837.dll and loads that
5. Next time, it will be .godot/libs/library-441902036275.dll

Main takeaway: Godot never directly loads path/to/library.dll, so the lock does not exist.

Other ideas:

• just move the file (may interfere with compiler)
• copy entire DLL into memory (needs more RAM; big waste if only a small part of the library is used)
• actively release the previously loaded library on events like focus-lost
  • might need hooks to notify the extension about unloading (i.e. solution to 2nd problem)

Is something like that realistic? Are file watches already implemented in Godot, e.g. for editor purposes?