quarkus: Kafka compressions Snappy and LZ4 not supported with native builds

This limitation is documented in #977 but not in https://quarkus.io/guides/kafka-guide. My question is what level of effort went in to trying to support snappy and gzip? Is JNI with native somehow ruled out, or might it be worth a try after #2412?

Describe the bug

Native builds disable support for message compression types Snappy and Gzip. Any attempt to consume or produce will result in an exception where the compression type is reported as an integer. The enumeration is in https://github.com/apache/kafka/blob/2.2.1/clients/src/main/java/org/apache/kafka/common/record/CompressionType.java.

A review comment in #977 documents the “hack” in https://github.com/quarkusio/quarkus/blob/0.16/extensions/kafka-client/runtime/src/main/java/io/quarkus/kafka/client/runtime/graal/SubstituteSnappy.java.

Expected behavior

Kafka consumers typically handle compression transparently. Compression is per message, not per topic. The broker enumerates supported compressions. Also messages are immutable and AFAIK there is no other way to remove compression than to rewrite the topic.

Actual behavior

For example when trying to consume a message compressed with Snappy:

java.lang.IllegalArgumentException: Unknown or unsupported compression type id: 2
	at org.apache.kafka.common.record.CompressionType.forId(CompressionType.java:61)
	at org.apache.kafka.common.record.DefaultRecordBatch.compressionType(DefaultRecordBatch.java:211)

I haven’t tested ZStandard yet. It isn’t explicitly disabled, but I think it too requires a native lib.

To Reproduce

Run https://quarkus.io/guides/kafka-guide
Produce using a non-quarkus kafka client with snappy compression.
The aforementioned exception occurs

Environment (please complete the following information):

Output of uname -a or ver: I’ve only run in docker.
Output of java -version: 8 and 11
GraalVM version (if different from Java): 1.0 rc15 and rc16 tested
Quarkus version or git rev: 0.14 through 0.16 tested

Workaround

Use a jvm build. Note that with the FROM in the generated Dockerfile (fabric8/java-alpine-openjdk8-jre) you must add RUN apk add --no-cache libc6-compat in order to avoid the error java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.7-3d2397f3-516f-45ab-aad1-94be731682f5-libsnappyjava.so: Error loading shared library ld-linux-x86-64.so.2: No such file or directory.

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 20 (12 by maintainers)

Most upvoted comments

Thanks @spike83! That’s a great step forward! I will have a look this week.

Here are a few thoughts:

The native library could be handled by the extension. If you have the snappy dependency, we can add it to the native image instructions.
Once we have a proper integration, we can remove the SubstituteSnappy, but with your work, sounds almost done! We can also have a conditional substitution checking if the snappy classes are on the classpath.

Which version of GraalVM did you use?

cescoffier on Feb 14, 2021

At the moment yes. I will be working on it soon.

cescoffier on Feb 16, 2021

I was able to make a workaround.

Therefore I removed https://github.com/quarkusio/quarkus/blob/master/extensions/kafka-client/runtime/src/main/java/io/quarkus/kafka/client/runtime/graal/SubstituteSnappy.java#L26-L65 and builded snapshot

In my project I added the snappy java dependency:

implementation group: 'org.xerial.snappy', name: 'snappy-java', version: '1.1.8.4'

In addition the native library snappyjava needs to be loaded by:

System.loadLibrary("snappyjava");

This native library file libsnappyjava.so has to be in the java.library.path which includes the /work directory of the docker container (ubi-minimal in my case), but it does not automatically go there.

Extract the snappy-java-x.y.z.jar and add the library by

COPY native/org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so /work/libsnappyjava.so
RUN chmod +x /work/libsnappyjava.so

The class org.xerial.snappy.SnappyInputStream.class has to be registered for reflection.

I would be happy to try to create a pr. But I would need some directions e.g. What would be the right way of adding the native libraries? How should this in general be approached?

Would it be an option to just remove the SubstituteSnappy class? One would get kind of helpful exceptions I think.

spike83 on Feb 13, 2021