confluent-kafka-dotnet: "InvalidOperationException: Failed to create thread" on Alpine Linux 3.7

Description

On Alpine Linux with .NET Core 2.1 and installed librdkafka consumer’s .ctor raises following exception:

Unhandled Exception: System.InvalidOperationException: Failed to create thread: No error information (0)
   at Confluent.Kafka.Impl.SafeKafkaHandle.Create(RdKafkaType type, IntPtr config)
   at Confluent.Kafka.Consumer..ctor(IEnumerable`1 config)
   at Confluent.Kafka.Consumer`2..ctor(IEnumerable`1 config, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer)
   at app.Program.Main(String[] args) in /app/Program.cs:line 9

Without installed librdkafka I get following exception:

Unhandled Exception: System.DllNotFoundException: Failed to load the librdkafka native library.
   at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)
   at Confluent.Kafka.Consumer..ctor(IEnumerable`1 config)
   at Confluent.Kafka.Consumer`2..ctor(IEnumerable`1 config, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer)
   at app.Program.Main(String[] args) in /app/Program.cs:line 9

How to reproduce

  1. docker build --tag repro-image . with following Dockerfile:
FROM microsoft/dotnet:2.1.301-sdk-alpine3.7

WORKDIR /app

# Install requirements
RUN apk add --update \
        bash \
        git \
        make \
        gcc \
        g++

# Build librdkafka from source
RUN git clone https://github.com/edenhill/librdkafka.git /librdkafka \
    && cd /librdkafka \
    && git checkout v0.11.4 \
    && ./configure \
    && make \
    && make install

# Create .NET Core console app
RUN cd /app \
    && dotnet new console \
    && dotnet add package -v 0.11.4 Confluent.Kafka
  1. docker run -it repro-image
  2. Fill Program.cs:
namespace app
{
    class Program
    {
        static void Main(string[] args)
        {
            var loaded = Confluent.Kafka.Library.Load("/usr/local/lib/librdkafka.so");
            System.Console.WriteLine("Library is loaded: " + loaded.ToString());
            var consumer = new Confluent.Kafka.Consumer<string, string>(
                new System.Collections.Generic.Dictionary<string, object>{{ "group.id", "test" }, { "debug", "all" }},
                new Confluent.Kafka.Serialization.StringDeserializer(System.Text.Encoding.UTF8),
                new Confluent.Kafka.Serialization.StringDeserializer(System.Text.Encoding.UTF8));
        }
    }
}
  1. dotnet run

Checklist

Please provide the following information:

  • Confluent.Kafka nuget version: 0.11.4
  • Operating system: Alpine Linux 3.7
  • Apache Kafka version:
  • Client configuration:
  • Provide logs (with “debug” : “…” as necessary in configuration)
  • Provide broker log excerpts
  • Critical issue

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

i’ve changed the labelling on this to ‘enhancement’ - we are considering adding out-of-the box support for alpine linux (given there is an official dotnet docker image based on this distro). This comes with the downside of increasing the size of librdkafka.redist however. Please +1 if you’d find this useful.

yes for sure. I don’t know if this’ll make it into 1.0 but it’s certainly a candidate for that.

@mhowlett do you still plan to add build of librdkafka for Alpine Linux in librdkafka.redist?

@mhowlett, I’ve just added python in apk add list and everything worked. It’s very strange, because just adding all of python deps didn’t help and Python itself doesn’t need to build librdkafka. After make install I could safely delete all these packages and both Consumer and Producer work as expected.

@edenhill, could you add build for Alpine in librdkafka.redist package? Here’s the Dockerfile to build:

FROM alpine:3.7

WORKDIR /librdkafka

# Install requirements
RUN apk add --no-cache --virtual .build-deps \
	bash \
	g++ \
	gcc \
	git \
	make \
	python \
# Optional deps to build with ssl, lz4:
	lz4-dev \
	cyrus-sasl-dev \
	openssl-dev

# Build librdkafka from source
RUN git clone https://github.com/edenhill/librdkafka.git . \
    && git checkout v0.11.4 \
    && ./configure \
    && make