confluent-kafka-dotnet: Kafka in Unity not working. Seems to have broken assemblies or conflicting with Unity's. How to fix?

Description

I want to integrate Kafka in Unity for a Windows build (I am on Windows).

However I either get errors such as this one:

Error: Could not load signature of Confluent.Kafka.IAsyncDeserializer`1[T]:DeserializeAsync due to: Could not load file or assembly 'System.Memory, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. assembly:System.Memory, Version=4.0.1.0,

Or if I install the System.Memory version that comes with the Kafka NuGet dependency, I get errors all over the place of Unity’s own method/class references no longer being found.

How to reproduce

I have tried installing the latest release (1.3.0) from nuget (using the UnityPackage ‘NuGetForUnity’). This did not seem to work and had the second type of error result described above.

I have then tried to build the source and copy the relevant DLL’s manually, and tried the same with the nuget release version. This gives the first type of error result described above.

Can anyone please help me with installing Kafka for Unity?

About this issue

Original URL
State: open
Created 4 years ago
Comments: 18 (2 by maintainers)

Links to this issue

Trying to implement Kafka in Unity, has anyone managed to get it working? I have issues with library / assembly versions conflicting. Could use some help!

Most upvoted comments

@ztolley I needed to do the same thing and recently figured it out. You may be missing the librdkafka.dll file for your system. For reference, here are the steps I used to get Kafka working in my project:

Download and install the NuGet for Unity .unitypackage from their GitHub.
In Unity, open “NuGet”->“Manage NuGet Packages” and install “Confluent.Kafka”.
From the NuGet package page for librdkafka, click “Download Package”.
Extract the archive. Under the “runtimes” folder, copy the folder for your system (I am on Ubuntu 18.04 so I picked “linux-x64”) and paste it under “Assets/Packages/Confluent.Kafka.<version>/lib/netstandard2.0” in Unity.
Create a C# script called “KafkaConsumer” and paste the code @Phantomb provided.
If needed, change BootstrapServers = "localhost:9092" to reflect your server and c.Subscribe("networktopic_data"); to the name of the topic you wish to subscribe to.
You might need to modify a few more things to get it to work for your case. To get started, find and replace StreamMessage with string for now. Replace StreamMessage message = ParseStreamMessage.Decode(cr.Value); with string message = Encoding.UTF8.GetString(cr.Value);. Replace ProcessPBMessages.Process(message); with Debug.Log(message);. Once you can play the scene and confirm you are receiving messages in the Console, you can try using Newtonsoft JsonConvert or something similar to work for your use case.

Hope that helps!

Mason-McGough on Dec 21, 2021

reopening, it would be good for us to test this and make this work ootb.

mhowlett on Oct 7, 2022

Hi Guys, Thank for this. I got Kafka working in Unity.

I’m using Unity 2020 with .Net 4.x

When I installed Confluent NuGet. I think most important is point 4 from the step-by-step guide.

NuGet

You can see my Packages Trees. you have to put all these file there. So download the librdkafka NuGet, change the extension to .zip or open it with zip extractor and go to Runtime -> your OS Version -> native -> pick all the file and bring it to Unity

knarfel on Oct 10, 2022

@ChulgooKim StreamMessage is just a struct I used to process / structure the sent data and was not relevant to the issue (which I fixed, see my latest comment.

Phantomb on Jul 20, 2020