llama_index: segmentation fault when using HuggingFaceEmbedding

Question Validation

  • I have searched both the documentation and discord for an answer.

Question

Hi,

I have this code that I throwing me the error:“segmentation fault”

import os
import streamlit as st

os.environ["REPLICATE_API_TOKEN"] = "my_token"

from llama_index.llms import Replicate

llama2_7b_chat = "meta/llama-2-7b-chat:8e6975e5ed6174911a6ff3d60540dfd4844201974602551e10e9e87ab143d81e"
llm = Replicate(
    model=llama2_7b_chat,
    temperature=0.01,
    additional_kwargs={"top_p": 1, "max_new_tokens": 300},
)

from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings import HuggingFaceEmbedding
from llama_index import ServiceContext

embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
service_context = ServiceContext.from_defaults(
    llm=llm, embed_model=embed_model
)

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(
    documents, service_context=service_context
)


#index = VectorStoreIndex.from_documents(documents)

# Get the query engine
query_engine = index.as_query_engine(streaming=True)

# Create a Streamlit web app
#st.title("LLM Query Interface")
query = st.text_input("Enter your query:")
submit_button = st.button("Submit")

if submit_button:
    # Query the engine with the defined query
    response = query_engine.query(query)
    st.write("### Query Result:")
    st.write(response)

if I disable embed_model code runs without problems.

I run the script via:

streamlit run my_script.py

I run this on Mac, Catalina, I have at least 250GB available memory. Please advise.

Thanks

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 21

Most upvoted comments

@logan-markewich could you please assist Ana with the specific issue she’s encountering related to privateGPT, including poor performance and a segmentation fault when querying simple questions from a file? Ana is looking for guidance from a developer on this matter. Thank you.

Details

Sources


About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

To address the issues you’re encountering with privateGPT, including poor performance and a segmentation fault when querying simple questions from a file, consider the following steps:

  1. Memory and Resources: Ensure your system has enough memory and resources. Segmentation faults often occur due to memory overuse.

  2. Driver Compatibility: Confirm that the updated drivers (version 535) are compatible with your system and the version of privateGPT you’re using.

  3. Request and Response Management: Investigate how requests and responses are managed, especially regarding data handling from attached files. Mismanagement can lead to performance issues and crashes.

  4. Configuration Parameters: Review and adjust configuration parameters like max_tokens, temperature, top_k, top_p, and repetition_penalty for optimal settings.

  5. Debugging: Use logging and debugging tools to trace the execution flow and pinpoint where the segmentation fault occurs.

  6. Dependencies: Ensure all dependencies are up to date and correctly installed. Issues with libraries like requests and sseclient-py can cause unexpected behavior.

  7. Simplified Scenario Testing: Run a simplified version of the operation without file attachments to determine if the problem is related to file handling.

  8. System Logs: Check system logs and error messages for clues about the cause of the segmentation fault.

  9. Fallback Configuration: If the issue persists, consider reverting to a previous stable driver update or privateGPT version while investigating.

For handling PDF files, privateGPT utilizes the PDFNougatOCR class for OCR technology to extract text. If your issue involves PDF file handling, reviewing this class and its methods might provide insights.

These steps are tailored to help troubleshoot and potentially resolve the issues you’ve described. If you need further assistance, please provide more specific details such as error messages, logs, or scenarios leading to the problems.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

🤖

Hello @anamariaUIC! I’m Dosu, a friendly bot here to assist you with your LlamaIndex issues while we wait for a human maintainer. I’m here to help you debug problems, answer questions, and guide you on your way to becoming a contributor. Let’s tackle this together!

The “segmentation fault” error you’re encountering is typically caused by a program trying to read or write an illegal memory location. In the context of your LlamaIndex code, it’s possible that the error is related to the HuggingFaceEmbedding and how it interacts with your system’s memory.

One potential issue could be the size of the model you’re trying to load with HuggingFace’s from_pretrained method. If the model is too large, it might be causing your program to exceed its allocated memory, leading to a segmentation fault.

Another potential issue could be the use of PyTorch’s torch.device in the infer_torch_device function. If you’re trying to use a GPU device that isn’t available or properly configured, it could lead to a segmentation fault.

Unfortunately, without more information about your system setup, the specific model you’re using, and the exact circumstances under which the error occurs, it’s difficult to provide a more specific solution.

If you’re able to provide more details, I’d be happy to help further. Otherwise, I’d recommend checking your system’s memory usage and your PyTorch device configuration as potential starting points for troubleshooting.

As for the role of the HuggingFaceEmbedding in the LlamaIndex codebase, it is responsible for generating embeddings for given text inputs using HuggingFace’s transformer models. The class has several methods for generating embeddings, including _embed, _get_query_embedding, _get_text_embedding, and _get_text_embeddings. It also has asynchronous versions of these methods.

I hope this helps! If you have any other questions or need further clarification, please don’t hesitate to ask.

Source Code

Sources


This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.