langchainjs: HNSWLib-node not found when using in a AWS Lambda function

I found the solution to this on Nextjs, user the following

/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,
    experimental: {
        appDir: true,
    },
    webpack: (config) => {
        config.externals = [...config.externals, "hnswlib-node"]

        return config
    },
}

export default nextConfig

I’m using Vercel serverless functions, which under the hood utilize AWS Lambda. Interestingly this used to work for me fine, but it stopped working when I switched over to NextJS 13 Route Handlers, which is currently in beta. I’m pretty sure the new NextJS 13 app router also utilizes Lambda, so I don’t know what changed compared to the old NextJS implementation specifically that is causing this to stop working.

If anyone else has this working with NextJS Route handlers, let me know.

Hi @joaotextor,

Thank you for your response. We appreciate your input. Since you’re no longer using it, we’ll go ahead and close the issue. If you ever encounter similar issues in the future, feel free to reach out. Thank you again!

Hi, @joaotextor,

I’m helping the langchainjs team manage their backlog and am marking this issue as stale. It looks like the issue involves the HNSWLib-node library not being found when the code is executed in an AWS Lambda function, despite being present in the node_modules folder. There were discussions about potential solutions, including the use of Docker to produce node_modules containing all the required libs, but it seems the error persisted for you and other users even after trying the suggested command. Some users found workarounds, while others are still seeking a resolution.

Could you please confirm if this issue is still relevant to the latest version of the langchainjs repository? If it is, please let the langchainjs team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!

@joaotextor try this this one works for me

import { MemoryVectorStore } from 'langchain/vectorstores/memory';

const embeddings = new OpenAIEmbeddings(); const store = await MemoryVectorStore.fromDocuments(docs, embeddings);

This is the alternate I have been using and it works fine for my use case.

// For creating a new vector from existing documents
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { OpenAIEmbeddings } from "langchain/embeddings/openai";

const apiKey = process.env.OPEN_AI_API_KEY;

const embeddings = new OpenAIEmbeddings({ openAIApiKey: apiKey });

// You can use one of the following method to create a vector store based on data available
const vectorStore = await MemoryVectorStore.fromDocuments(docs, embeddings);
// or
const vectorStore = await MemoryVectorStore.fromTexts(textArray, metadataArray, embeddings);

// Once it is created you can persist the data as json file either in S3 or any other persistent storage
const jsonToStore = vectorStore.memoryVectors; // I saved this data as json file

// Next time when needed I load this file again using the following code.
const vectorStoreData = loadDataFromJsonFileStoredAbove();
const vectorStore = await MemoryVectorStore.fromExistingIndex(embeddings);
vectorStore.memoryVectors = vectorStoreData;

// Now do a similarity search 
const result = await vectorStore.similaritySearch(query, 4);

Any chance of some more detail on how to do this for a docker newbie? Essentially this doesn’t work for building my lambda layer because most of the node modules require 18.x.

So, can I “docker pull node” to get the latest version, then do some “docker run docker run --rm -v “$PWD”:/var/task nodes npm install langchain hnswlib-node my_other_modules” magic to create a node_modules that I can zip up and use as a lambda layer?

Thanks for any help,

David

@manduks Cheers mate! After a few days of headache with this issue, your solution worked for me as charm. Awesome stuff! Thank you again!