llama_cloud_services: LLamaparse could not extract figures from one pdf file despite good pdf quality (edge-case)

Describe the bug I am trying to extract graphical figures and text in some pdf files. The code below, which is inspired by a helpful response of @tkcoding to another issue does work for many pdfs with figures but there is an edge case, that I can not parse despite some “prompt” engineering effort:

This is the code I used, that is mostly adopted from (https://github.com/run-llama/llama_parse/issues/317)

from llama_parse import LlamaParse
from llama_index.core import StorageContext
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import MarkdownElementNodeParser
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding


from llama_index.core.schema import ImageDocument
from typing import List
from llama_index.core.node_parser import LlamaParseJsonNodeParser
from llama_index.core.schema import BaseNode, TextNode, Document
from llama_index.core.tools import QueryEngineTool, ToolMetadata
from llama_index.core.query_engine import SubQuestionQueryEngine
import glob
import nest_asyncio
import os
from dotenv import load_dotenv

load_dotenv()
nest_asyncio.apply()


ins = """
You are a highly proficient language model designed to convert pages from PDF into structured markdown text. Your goal is to accurately transcribe text, and identify and describe images and figures, particularly graphs and other graphical elements.

You have been tasked with creating a markdown copy of each page from the provided PDF. Each image description must include a full description of the content, a summary of the graphical object.

Maintain the sequence of all the elements.

For the following element, follow the requirement of extraction:
for Text:
   - Extract all readable text from the page.
   - Exclude any diagonal text, headers, and footers.

for Text which includes hyperlink:
    -Extract hyperlink and present it with the text
    
for Image Identification and Description:
   - Identify all images, graphs, and other graphical elements on the page.
   - If image contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing.
   - For each image, include a full description of the content in the alt text, followed by a brief summary of the graphical object.
   - If the image has a subtitle or caption, include it in the description.
   - If the image has a organisation chart , convert it into a hierachical understandable format.
   - for graph , extract the value in table form as markdown representation

for Figure Identification and Description:
   - Identify all Figures, graphs, and other graphical elements on the page. Some of those are created using alphabets, so be creative
   - If figure contains wording that is hard to extract , flag it with <unidentifiable section> instead of parsing.
   - For each figure, include a full description of the content in the alt text, followed by a brief summary of the graphical object.
   - If the figure has a subtitle or caption, include it in the description.
   - If the figure has a organisation chart , convert it into a hierachical understandable format.
   - for graph , extract the value in table form as markdown representation



    
# OUTPUT INSTRUCTIONS

- Exclude any diagonal text, headers, and footers from the output.
- For each figure, image and graph, provide a detailed description and summary.
"""

class llama_document_parser(object):
    def __init__(self,parsing_ins):
        self.parser = LlamaParse(
            verbose=True,
            ignore_errors=False,
            invalidate_cache=True,
            do_not_cache=True,
        )

    def get_image_text_nodes(self,download_path: str,json_objs: List[dict]):
        """Extract out text from images using a multimodal model."""
        image_dicts = self.parser.get_images(json_objs, download_path=download_path)
        image_documents = []
        img_text_nodes = []
        for image_dict in image_dicts:
            image_doc = ImageDocument(image_path=image_dict["path"])
            img_text_nodes.append(image_doc)
        return img_text_nodes

    def document_processing_llamaparse(self,file_name: str ,image_output_folder:str):
        """Parse document in using llamaparse and return extracted elements in json format"""
        json_objs = self.parser.get_json_result(file_name)
        json_list = json_objs[0]["pages"]
        print(json_list)
        if not os.path.exists(image_output_folder):
            os.mkdir(image_output_folder)

        image_text_nodes = self.get_image_text_nodes(image_output_folder,json_objs)
        return json_list

llama_parser = llama_document_parser(parsing_ins=ins)
# llamaparse to extract documents
json_list = llama_parser.document_processing_llamaparse(file_name="/content/drive/MyDrive/Omdena_Challenge/new_LK_tea_dataset/Guideline/with_Figure/TRISL_Guideline_02_2021e_Aug2021Intercropping.pdf",
                              image_output_folder="image_test")

Files TRISL_Guideline_02_2021e_Aug2021Intercropping.pdf

Job ID 9d32683a-a45e-400b-8099-4f50ab29881f

Client:

  • Python Library
  • API
  • Frontend (cloud.llamaindex.ai)
  • Notebook

Additional context I am trying to extract Figures.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

I see. Out of the box, we don’t detect it as an image (because it’s not really an image) but we just released a layout detection option: https://docs.cloud.llamaindex.ai/llamaparse/features/layout_extraction It’s 1 extra credit per page but it may solve your problem.

@tkcoding There’s no images in the document. It’s texts representing some charts and diagrams. The OCR won’t help.

@haniehm On the figure where there’s a lot of T T T T T T T T T T T T T T T T T T T T T T T, the markdown from the response shows:

Figure 1
**Description:** The figure illustrates the spatial arrangement of tea and coconut in an intercropping system. It shows the layout with specific measurements indicated for the spacing between the plants. The letters represent different types of plants:
- C - Coconut
- T - Tea
- G - Gliricidia

I don’t really know what you expect the output to be but it seems to fit your prompt.

Furthermore, you can use the Premium Mode for free. It just cost more credits. Here your example in Markdown: TRISL_Guideline_02_2021e_Aug2021Intercropping-1.pdf.md

@haniehm Sorry for the last reply , if you haven find the solution. Basically llama is still working on improving the image extraction ,specifically for images that looks like a bunch of text in an images often confuse the outcome of the extraction (extracted as bunch of text with delimiter).

Solution: Hybrid with other framework like paddleOCR (best solution) or pymupdf (OK solution). To extract the images. Create a multi-vector retriever with images processed into text and taking markdown from llamaparse.

This will yield the best quality so far.

Note: Above paddleOCR and pymupdf are free as well so it fits your project purpose.