com.unity.netcode.gameobjects: How to trigger NGO CodeGen via custom buildpipeline?
This is a long shot and I’m assuming/guessing things based on what I have researched. I also know I’m pushing the limits here - but hey somebody have to do it 😃
Background
The background for this is that my game supports mods and I’m currently working on adding multiplayer to my game using NGO.
So far things are going pretty good - but there is one major issue I still haven’t figured out and it’s a really big deal as mod creators will be very limited if I can’t get this to work.
I’m using UMod to help support mods in my game (https://assetstore.unity.com/packages/tools/integration/umod-2-0-58293). As far as I understand, it adds feature on top and around Asset Bundles.
The way UMod work is pretty nice.
As the developer of the game (me) I setup various things, like what scripts should be included in the “Mod Tools” and other settings. I then build the “Mod Tools” for my game from inside Unity. This result in a .unitypackage file.
As the mod creators (people from the community) you then open up a fresh Unity and import this .unitypackage file (Mod Tools) and get some Unity windows to create a “mod” and can then build a mod - which can include custom scripts.
The mod creator then “build mod” from a custom Unity menu made by UMod, which generates a .mod file that the game can then pick up. At least simply put - the important part here is the UMod builds the mod here. Remember that.
The problem
So, I have discovered that ClientRPCs, ServerRPCs and NetworkVariables are NOT working if a modder is using it in his custom scripts.
I have done some research that indicates (and here comes the guessing part) that NGO is doing some CodeGen at “game build time” - I guess here: https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/blob/develop/com.unity.netcode.gameobjects/Editor/CodeGen/NetworkBehaviourILPP.cs
I’m guessing that it doesn’t work, because this CodeGen is not happening on the custom scripts in the mod when it’s build by UMod.
UMod supports custom BuildProcessors to run custom code when building the mods like this:
using System.Collections.Generic;
using UMod.BuildEngine;
using UMod.BuildPipeline;
using UMod.BuildPipeline.Build;
using UMod.Shared.Resources;
using UnityEngine;
// Add a build processor for script assets since we are only interested in scripts that will be compiled by uMod.
// High priority means that it will run after script compilation so build assemblies collection will already be filled out with compiled assemblies.
[UModBuildProcessor(".cs", 1000, true)]
public class PatchAssembliesExample : BuildEngineProcessor
{
// Methods
// Called by the build engine just after compilation if the mod includes any script content, passing all script assets as a parameter (Not interested in those).
public override void ProcessAssetBatch(BuildContext context, IEnumerable<BuildPipelineAsset> assets)
{
// get assemblies
ModAssemblyEntry[] entries = context.BuildAssemblies.Assemblies;
// Patch all
foreach(ModAssemblyEntry entry in entries)
{
PatchAssembly(context, entry);
}
}
private void PatchAssembly(BuildContext context, ModAssemblyEntry entry)
{
// Name info
string asmName = entry.AssemblyName;
// Get assembly image - loadable by 'Assembly.Load'
byte[] asmImage = entry.AssemblyImage;
// TODO - Run custom patcher code
// Get the patched result
byte[] asmImagePatched = ...;
// Check for patched
if (asmImagePatched != null)
{
// Remove original assembly
context.BuildAssemblies.RemoveFromBuild(entry);
// Register new assembly
if (context.BuildAssemblies.RegisterAssemblyForBuild(asmImagePatched, true) == false)
Debug.LogWarning("Failed to register patches assembly for build! : " + asmName);
}
}
}
So, here comes the million dollar question that will make the sky the limits for modders of my game - how can I trigger this CodeGen for the scripts in the mod?
Note that NGO is installed as a dependency to the Mod Tools, so NGO is installed in the mod creators Unity project.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 54
Same issue here, i would really like to be able to invoke thoses postprocessors but i see that some others peoples ran into that issue as well soo can’t wait for that feature.
Oh this is awesome! We got into this issue as well and was wondering whether or not to wait for an update. The looks promising! Thanks!
Hello @NoelStephensUnity am just here to see if there is been some progress on our issue, i see that 2.0.0-exp5 got out recently and i would like to know if our issue is in that 2.0.0 queue ?
@guanaco0403 @Laumania Sorry, just finished getting both the NGO v2.0.0-exp.2 published and NGO v1.9.1 should be showing up this week (in is in final stage prior to publication). Kitty migrated over to the Transport team so this lands on my plate.
I will add this to one of my queues…which I might place that in the NGO v2.0.0 queue to allow for more flexibility in what I can change.
Just following up to provide some visibility on #2813 (what Kitty has been working on that might be of interest for those who have posted here). There are still more “things in progress” that we are working through and pushing out to help improve some long-standing areas in NGO.
@ShadauxCat will circle back to take a look at this once she is done doing super-awesome things! :godmode:
Apologies for not responding sooner. We have been working on various areas of NGO and this is still something we have in our queue. Since we will have to re-create a usage scenario it adds to the complexity.
Of course, NGO is open source and so if it is a true blocker then forking and making adjustments is always an option (and it would provide us with a better use case to target).
Unfortunately, we most likely will not be able to get to this until after the update that follows v1.8.0.
@scottyboy805 Ahhh, so that makes more sense… user specific mod with scripts that might implement either @Laumania’s game’s netcode classes and/or make their own classes will not be able to add additional unique netcode behaviors (i.e. RPCs, NetworkVariables, or anything else requiring ILPP) because of how the scripts are compiled.
So, I am thinking the only real way to do that is to be able to invoke the postprocessors after you have compiled the scripts which would just require you to have access to all of the internal classes here: https://github.com/Unity-Technologies/com.unity.netcode.gameobjects/tree/develop/com.unity.netcode.gameobjects/Editor/CodeGen
I will noodle over this a bit more, but I think exposing some of the CodeGen classes could provide enough to possibly make it possible to handle this in the PatchAssembliesExample.PatchAssembly method… The post processors basically need something like:
Need to look at this side of things a bit more…
Hi @NoelStephensUnity
This makes total sense. Of cause I would be happy if you just fixed this issue, but I completely understand that this scenario is a more edge-case and not something the majority of Unity games have a need for.
Even though I’m a solo developer, I am lucky so have some super engaged and skilled modders in my community, so I think I will try and work with them and see if we can make it work in a fork of the NGO project. If we find a simple enough solution I will bring it back to you here.
Again, even though its really annoying you cannot fix this right now, I really appreciate you (Unity) take the time to go back, refactor and improve things as its so important in the long run, even though it might be a bit boring right now.
Appreciate your reply, even though we would maybe have liked to have it a bit sooner, but I understand.
Lets keep this issue open, so I/we can get back with a solution if we find one.
FYI @guanaco0403 @Keltusar
still nothing @NoelStephensUnity ? it seems verry silent lately
Oh, hope you and your family are all good!
About v1.x, this doesn’t have to be a v2.0 feature, as I plan to upgrade to v2.0 anyway - so if it’s easier to do in v2.0 just enable it there I guess 😃
REALLY good to hear back from you! We appreciate it!
Hi @NoelStephensUnity Any news or progress on this one? 😃
@NoelStephensUnity We understand, thanks for your reply! 😃
Any rough ETA on when NGO v1.8.0 will be release, aka when you will start looking into this, just roughly?
Hi @NoelStephensUnity
Anything new on this? It’s becoming more and more limiting for my modders that they can’t do this 😃
@Laumania Yeah looks like it will make life a lot easier if this can be added. Reflection is an idea (although a fragile one) but I don’t think it can work in this case since it looks like the API will require an implementation of
ICompiledAssembly, which can’t be done from reflection, so some public API will need to be added I would think to get this all working nicley.@scottyboy805 @NoelStephensUnity I’m so happy to see that this might be possible - as said - the sky will be the limit if my mod community can use RPCs and NetworkVariables 😄
@scottyboy805 If we knew what method to call during the UMod build pipeline, couldn’t we maybe use reflection to call it - just a a way to check if it actually works?
But of cause, some more bulletproof solution is needed.
Really excited about this progress! 😄
@NoelStephensUnity Awesome, yes exposing some of the internal classes would solve the problem if that is not too much hassle. Certainly implementing something like the above interface will be ideal and will be easy for us or even developers themselfs to extend, since uMod already allows access to such information in a build event. Alternativley if it is too much work to expose some of the API or would break the intended usage etc, then even just a simple helper method like
byte[] CodeGenUtil.PatchExternalAssembly(byte[] asmImage)could work for this purpose just as an example, but what ever you think if best.Thanks for all the help.
Hi @NoelStephensUnity, My name is Scott, co founder of Trivial Interactve who develop the uMod 2.0 modding asset mentioned here. Just wanted to offer some insight into what uMod is doing as part of its compile process and maybe you will have some more ideas with a better understanding of what is happening.
The first point to mention is that uMod does not actually patch any assemblies or game code as such. Instead it supports additive modding which means compiling additional script assemblies that work as add-ons to the game. To do this uMod runs the Roslyn compiler manually when building mods to compile the .csproj files for the mod project, and I guess since this step is done manually and not via the Unity build pipeline, the code gen is not running on those assemblies which is to be expected.
Since uMod offers extensive build events and callback, one solution that could work is to run the code gen manually just after uMod has compiled the scripts into an assembly. That was the sample code above I provided to @Laumania which is a uMod build event, but obviously we don’t know how to run the code gen manually yet. I had a quick look at the code gen source and it looks to me like it is probably not designed to be run manually from some source other than the Unity build pipeline. Is that the case or are we missing some public API that we can use to trigger this process? If we can get to run the code gen manually then I think it will be problem solved and issue closed, but I suspect it may not be that easy? And just to be clear, that would need to be invoked on an already compiled managed assembly (.dll). I see there is an
ICompiledAssemblyinterface which I guess can take care of that, but again it is all internal and probably not designed to be used by the likes of us.I appreciate what you say about abstracting the networking aspect of the code base away so that modders don’t need to deal with that at all. It does seems like a suitbale solution but my only concerns would be the amount of extra work that would mean for game developers. Ie. needing to desing an API to abstract away implementation details of the networking while making sure the API is feature complete and usable in the real world. Seems to me to be quite a complex task although it would mean that modders don’t need to learn/work with netcode directly which is probably a plus point, and developers could switch networking backends without issue. Pehaps a quick example of such an abstracting layer would be helpful if you have time just to see what sort of thing we are ataling about. I guess it is probably not as simple as modders marking thier code with RPC attributes or anything like that, and instead will need to work directly with the API to send RPC’s and serialized data, so a different more manualy sort of setup I would assume.
Anyway, hopefully that extra info can be somewhat helpful to provide a nice solution to this unconevntional issue/use case. Regards, Scott.
@Laumania Hello! 👋 Hmmm… I am sure it is an order of operations type of issue… The one thing I am not 100% clear on is whether the user creating the mod is adding additional NetworkVariables and RPCs =or= if they are just extending your classes that already have the NetworkVariables and RPCs and when they extend/mod certain classes that contain NetworkVariables (most likely with custom serializable types) and RPCS those NetworkVariables and RPCs no longer function correctly.
If the later scenario is the case, then this is most likely what is happening:
So…unless you are trying to preserve already existing mods (most likely is too late for already UMod generated patches), the best way to avoid losing the ILPP generated code is to create a full abstraction layer between the classes the users will modify and the classes that contain the RPCs and NetworkVariables:
Your Project’s Primary Netcode Class (in primary Assembly/Assemblies) --> i.e. PrimaryAssembly.asmdef
Your Project’s Customizable Classes (in secondary Assembly/Assemblies)–> i.e. CustomAssembly.asmdef
You would then most likely need to create some form of generic interface that handles the specific behavior(s) you want to allow users to override/modify…so the
CustomizableClasswould implement something likeISomeCoreNetcodeClassActionsthat would then register itself with the referencedSomeCoreNetcodeClasscomponent…then when specific “customizable” actions need to be invoked theSomeCoreNetcodeClasswould invoke the registeredISomeCoreNetcodeClassActions.Where
SomeCoreNetcodeClassalready has a “default”ISomeCoreNetcodeClassActionsimplementation in the event nothing is “customized”.Basically you need to only allow UMod to be able to modify assembly/assemblies that only have references to classes in your core netcode enabled assemblies but not modify the assemblies that contain those classes…thus the reason for having different assemblies… one with the compiled NGO derived classes…and one with just the interface implementations (i.e. the default) and the classes that register those implementations with the core netcode enabled classes.
This way UMod will only replace the “default interface implementation” assemblies and not the assemblies with the injected code.
I am a bit heads down currently working on some more updates for NGO, but if you don’t hear back from me by the end of this week just ping me and I will try to provide you with a very simple skeleton project of what I am talking about.