ml-commons: [BUG] unable to run knn search with neural query on OS 2.16.0

What is the bug? Searching with neural query brings down OS 2.16.0.

This is happening in OS 2.16.0 Image FROM opensearchproject/opensearch:2.16.0 but not in 2.15.0 or lower

How to reproduce the bug?

  1. Upload and deploy ML model
We have a custom transformers model but you can upload another one with:

POST /_plugins/_ml/models/_register
{
  "name": "huggingface/sentence-transformers/msmarco-distilbert-base-tas-b",
  "version": "1.0.2",
  "model_group_id": "Z1eQf4oB5Vm0Tdw8EIP2",
  "model_format": "TORCH_SCRIPT"
}

POST /_plugins/_ml/models/dxyObJEBGnTvwYNln7p8/_deploy

  1. Create index with settings knn = true and an ingest pipeline for the model
PUT _ingest/pipeline/test
{
  "description": "",
  "processors": [
    {
      "text_embedding": {
        "model_id": "dxyObJEBGnTvwYNln7p8",
        "field_map": {
          "text": "text_embedding"
        }
      }
    }
  ]
}

PUT /testing
{
  "settings": {
    "index.knn": true,
    "index.default_pipeline": "test"
  },
  "mappings": {
    "properties": {
      "text": {
        "type": "text"
      },
      "text_embedding": {
        "type": "knn_vector",
        "dimension": 2
      }
    }
  }
}

  1. ingest some docs POST /testing/_doc
{
  "text": "testing knn"
}
  1. Search with a neural query
POST /testing/_search
{
  "query": {
    "neural": {
      "text_embedding": {
        "model_id": "dxyObJEBGnTvwYNln7p8",
        "query_text": "testing_neural"
      }
    }
  }
}
  1. Cluster goes down with:
opensearch-node2       | fatal error in thread [opensearch[opensearch-node2][refresh][T#3]], exiting
opensearch-node2       | java.lang.UnsatisfiedLinkError: /usr/share/opensearch/plugins/opensearch-knn/lib/libopensearchknn_common.so: /usr/share/opensearch/data/ml_cache/pytorch/1.13.1-cpu-precxx11-linux-x86_64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /usr/share/opensearch/plugins/opensearch-knn/lib/libopensearchknn_util.so)
opensearch-node2       | 	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
opensearch-node2       | 	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331)
opensearch-node2       | 	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197)
opensearch-node2       | 	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139)
opensearch-node2       | 	at java.base/jdk.internal.loader.NativeLibraries.findFromPaths(NativeLibraries.java:259)
opensearch-node2       | 	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:251)
opensearch-node2       | 	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2451)
opensearch-node2       | 	at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
opensearch-node2       | 	at java.base/java.lang.System.loadLibrary(System.java:2063)
opensearch-node2       | 	at org.opensearch.knn.jni.JNICommons.lambda$static$0(JNICommons.java:26)
opensearch-node2       | 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
opensearch-node2       | 	at org.opensearch.knn.jni.JNICommons.<clinit>(JNICommons.java:25)
opensearch-node2       | 	at org.opensearch.knn.index.codec.transfer.VectorTransferFloat.transfer(VectorTransferFloat.java:68)
opensearch-node2       | 	at org.opensearch.knn.index.codec.transfer.VectorTransferFloat.close(VectorTransferFloat.java:59)
opensearch-node2       | 	at org.opensearch.knn.index.codec.util.KNNCodecUtil.getPair(KNNCodecUtil.java:61)
opensearch-node2       | 	at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.addKNNBinaryField(KNN80DocValuesConsumer.java:147)
opensearch-node2       | 	at org.opensearch.knn.index.codec.KNN80Codec.KNN80DocValuesConsumer.addBinaryField(KNN80DocValuesConsumer.java:87)
opensearch-node2       | 	at org.apache.lucene.index.BinaryDocValuesWriter.flush(BinaryDocValuesWriter.java:132)
opensearch-node2       | 	at org.apache.lucene.index.IndexingChain.writeDocValues(IndexingChain.java:424)
opensearch-node2       | 	at org.apache.lucene.index.IndexingChain.flush(IndexingChain.java:282)
opensearch-node2       | 	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:445)
opensearch-node2       | 	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:496)
opensearch-node2       | 	at org.apache.lucene.index.DocumentsWriter.maybeFlush(DocumentsWriter.java:450)
opensearch-node2       | 	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:643)
opensearch-node2       | 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:578)
opensearch-node2       | 	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)
opensearch-node2       | 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:355)
opensearch-node2       | 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:345)
opensearch-node2       | 	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:112)
opensearch-node2       | 	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:170)
opensearch-node2       | 	at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:72)
opensearch-node2       | 	at org.opensearch.index.engine.OpenSearchReaderManager.refreshIfNeeded(OpenSearchReaderManager.java:52)
opensearch-node2       | 	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
opensearch-node2       | 	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:240)
opensearch-node2       | 	at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:433)
opensearch-node2       | 	at org.opensearch.index.engine.InternalEngine$ExternalReaderManager.refreshIfNeeded(InternalEngine.java:413)
opensearch-node2       | 	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:167)
opensearch-node2       | 	at org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:213)
opensearch-node2       | 	at org.opensearch.index.engine.InternalEngine.refresh(InternalEngine.java:1865)
opensearch-node2       | 	at org.opensearch.index.engine.InternalEngine.maybeRefresh(InternalEngine.java:1844)
opensearch-node2       | 	at org.opensearch.index.shard.IndexShard.scheduledRefresh(IndexShard.java:4648)
opensearch-node2       | 	at org.opensearch.index.IndexService.maybeRefreshEngine(IndexService.java:1157)
opensearch-node2       | 	at org.opensearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:1301)
opensearch-node2       | 	at org.opensearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:159)
opensearch-node2       | 	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882)
opensearch-node2       | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
opensearch-node2       | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
opensearch-node2       | 	at java.base/java.lang.Thread.run(Thread.java:1583)

What is the expected behavior? Dont crash and perform a search

What is your host/environment? Docker with FROM opensearchproject/opensearch:2.16.0 Image

NOTE: This issue only happens the first time you run a neural query. After running a neural query sometime after the first one, the cluster seems to get the libraries. I do not know what was changed from OS 2.15 to 2.16

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 30 (9 by maintainers)

Most upvoted comments

@yuye-aws this error looks specific to native knn engines. Can you check which one you’re using, if it’s one of native ones (faiss or nmslib) can you try defining lucene as your knn engine.