confluent-kafka-dotnet: "InvalidOperationException: Failed to create thread" on Alpine Linux 3.7
Description
On Alpine Linux with .NET Core 2.1 and installed librdkafka consumer’s .ctor raises following exception:
Unhandled Exception: System.InvalidOperationException: Failed to create thread: No error information (0)
at Confluent.Kafka.Impl.SafeKafkaHandle.Create(RdKafkaType type, IntPtr config)
at Confluent.Kafka.Consumer..ctor(IEnumerable`1 config)
at Confluent.Kafka.Consumer`2..ctor(IEnumerable`1 config, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer)
at app.Program.Main(String[] args) in /app/Program.cs:line 9
Without installed librdkafka I get following exception:
Unhandled Exception: System.DllNotFoundException: Failed to load the librdkafka native library.
at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)
at Confluent.Kafka.Consumer..ctor(IEnumerable`1 config)
at Confluent.Kafka.Consumer`2..ctor(IEnumerable`1 config, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer)
at app.Program.Main(String[] args) in /app/Program.cs:line 9
How to reproduce
docker build --tag repro-image .with following Dockerfile:
FROM microsoft/dotnet:2.1.301-sdk-alpine3.7
WORKDIR /app
# Install requirements
RUN apk add --update \
bash \
git \
make \
gcc \
g++
# Build librdkafka from source
RUN git clone https://github.com/edenhill/librdkafka.git /librdkafka \
&& cd /librdkafka \
&& git checkout v0.11.4 \
&& ./configure \
&& make \
&& make install
# Create .NET Core console app
RUN cd /app \
&& dotnet new console \
&& dotnet add package -v 0.11.4 Confluent.Kafka
docker run -it repro-image- Fill Program.cs:
namespace app
{
class Program
{
static void Main(string[] args)
{
var loaded = Confluent.Kafka.Library.Load("/usr/local/lib/librdkafka.so");
System.Console.WriteLine("Library is loaded: " + loaded.ToString());
var consumer = new Confluent.Kafka.Consumer<string, string>(
new System.Collections.Generic.Dictionary<string, object>{{ "group.id", "test" }, { "debug", "all" }},
new Confluent.Kafka.Serialization.StringDeserializer(System.Text.Encoding.UTF8),
new Confluent.Kafka.Serialization.StringDeserializer(System.Text.Encoding.UTF8));
}
}
}
dotnet run
Checklist
Please provide the following information:
- Confluent.Kafka nuget version: 0.11.4
- Operating system: Alpine Linux 3.7
- Apache Kafka version:
- Client configuration:
- Provide logs (with “debug” : “…” as necessary in configuration)
- Provide broker log excerpts
- Critical issue
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (8 by maintainers)
i’ve changed the labelling on this to ‘enhancement’ - we are considering adding out-of-the box support for alpine linux (given there is an official dotnet docker image based on this distro). This comes with the downside of increasing the size of
librdkafka.redisthowever. Please +1 if you’d find this useful.yes for sure. I don’t know if this’ll make it into 1.0 but it’s certainly a candidate for that.
@mhowlett do you still plan to add build of librdkafka for Alpine Linux in
librdkafka.redist?@mhowlett, I’ve just added
pythoninapk addlist and everything worked. It’s very strange, because just adding all of python deps didn’t help and Python itself doesn’t need to build librdkafka. Aftermake installI could safely delete all these packages and bothConsumerandProducerwork as expected.@edenhill, could you add build for Alpine in librdkafka.redist package? Here’s the Dockerfile to build: