snappy-java: GraalVM: Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64

Hello team,

This is my first issue in this repo. Is not anything, big thanks for this very cool project.

I would like to reach out in order to report an issue: My project is a GraalVM native image project, a simple Kafka consumer.

In my dependencies, I do have the most recent Snappy:

        <dependency>
            <groupId>org.xerial.snappy</groupId>
            <artifactId>snappy-java</artifactId>
            <version>1.1.9.1</version>
        </dependency>

On my machine, Ubuntu, I do have Snappy installed.

root@dlp:~# apt -y install snapd
root@dlp:~# systemctl  enable snapd.service snapd.socket
root@dlp:~# source /etc/profile.d/apps-bin-path.sh

Yet, when running the application, I am facing this 100% reproducible issue:

Caused by: org.apache.kafka.common.KafkaException: Received exception when fetching the next record from my-topic-1. If needed, please seek past the record to continue consumption.
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1676)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.access$1900(Fetcher.java:1497)
	at org.apache.kafka.clients.consumer.internals.Fetcher.fetchRecords(Fetcher.java:717)
	at org.apache.kafka.clients.consumer.internals.Fetcher.collectFetch(Fetcher.java:683)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1314)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1243)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
	at reactor.kafka.receiver.internals.ConsumerEventLoop$PollEvent.run(ConsumerEventLoop.java:357)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base@17.0.7/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base@17.0.7/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base@17.0.7/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base@17.0.7/java.lang.Thread.run(Thread.java:833)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
	at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
Caused by: org.apache.kafka.common.KafkaException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
	at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:46)
	at org.apache.kafka.common.record.CompressionType$3.wrapForInput(CompressionType.java:94)
	at org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:276)
	at org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:280)
	at org.apache.kafka.common.record.DefaultRecordBatch.streamingIterator(DefaultRecordBatch.java:364)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.nextFetchedRecord(Fetcher.java:1619)
	at org.apache.kafka.clients.consumer.internals.Fetcher$CompletedFetch.fetchRecords(Fetcher.java:1656)
	... 16 common frames omitted
Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=x86_64
	at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:343)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:177)
	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:155)
	at org.xerial.snappy.Snappy.init(Snappy.java:70)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
	at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:145)
	at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:99)
	at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:59)
	at org.apache.kafka.common.compress.SnappyFactory.wrapForInput(SnappyFactory.java:44)
	... 22 common frames omitted

Could you please help on this issue?

About this issue

Original URL
State: open
Created a year ago
Reactions: 1
Comments: 15 (3 by maintainers)

Most upvoted comments

Thanks a lot @liuxgo it works fine with fixes you suggested.

<plugin>
	<groupId>org.graalvm.buildtools</groupId>
	<artifactId>native-maven-plugin</artifactId>
	<configuration>
		<buildArgs>
			<arg>-H:ResourceConfigurationFiles=${project.basedir}/src/main/resources/META-INF/native-image/resource-config.json</arg>
			<arg>-H:Log=registerResource:3</arg>
		</buildArgs>							
	<requiredVersion>22.3</requiredVersion>
	</configuration>
	...
</plugin>

{
  "bundles": [],
  "resources": {
    "includes": [
      {  
        "condition": {
                    "typeReachable": "org.xerial.snappy.Snappy"
                }, 
        "pattern": "org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so"
      }
    ]
  }
}

Build log:

[thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so

pavelorehov on Oct 5, 2023

@pavelorehov Maybe you can try to modify resource-config.json like below:

add contidion
delete first “/” of pattern

      "condition": {
          "typeReachable": "org.xerial.snappy.Snappy"
        },
        "pattern": "org/xerial/snappy/native/Windows/[a-z_0-9]+/snappyjava.dll"

And you can add native maven build args <buildArg>-H:Log=registerResource:3</buildArg>, it will show register resource or not.

  [thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Windows/x86/snappyjava.dll
  [thread:1] scope: main.registerResource
  ResourcesFeature: registerResource: org/xerial/snappy/native/Windows/x86_64/snappyjava.dll

liuxgo on Oct 3, 2023

@patpatpat123 Write resource-config.json like this, I only need binary file for Windows.

{
  "bundles": [],
  "resources": {
    "includes": [
      {
        "condition": {
          "typeReachable": "org.xerial.snappy.Snappy"
        },
        "pattern": "org/xerial/snappy/native/Windows/[a-z_0-9]+/snappyjava.dll"
      }
    ]
  }
}

Add build args in pom.xml like this: <buildArg>-H:ResourceConfigurationFiles=native-image/resource-config.json</buildArg>

liuxgo on Sep 25, 2023

It seems sqlite-jdbc, which is using the same mechanism for loading native library at runtime, can support GraalVM https://github.com/xerial/sqlite-jdbc#graalvm-native-image-support

Probably we can borrow the same technique

xerial on Jul 15, 2023

@patpatpat123 I just verified that this is actually quite easy to do at build-time: just include -H:IncludeResources=".*/libsnappyjava.dylib" in the native-image call and make sure that the native lib is the correct one for your platform (and of course provide the snappy jar in the classpath for native-image)

so the complete call would look like this native-image -cp [your_classes]:[PATH]/snappy-java-1.1.9.1.jar -H:IncludeResources=".*/native-lib" MainClass

inspect the snappy-jar to find the native libs

@xerial this is not something that can be fixed/changed from the library side

shautvast on Jun 21, 2023