Scrapegraph-ai: [1.14.0+] pydantic ValidationError with SmartScraperGraph

Describe the bug Since version 1.14.0 (tested here on 1.15.0), SmartScraperGraph on OpenAI stopped working with a pydantic-related error:

/venv/lib/python3.12/site-packages/google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported, `google-crc32c` is using a pure python implementation that is significantly slower. If possible, please configure a c build environment and compile the extension                                                                                        warnings.warn(_SLOW_CRC32C_WARNING, RuntimeWarning)
--- Executing Fetch Node ---
--- (Fetching HTML from: https://perinim.github.io/projects/) ---
--- Executing Parse Node ---
--- Executing GenerateAnswer Node ---
Traceback (most recent call last):
  File "/llm_scraper/test_bug.py", line 50, in <module>                                                                                                        result = smart_scraper_graph.run()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/smart_scraper_graph.py", line 114, in run                                                      self.final_state, self.execution_info = self.graph.execute(inputs)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 263, in execute                                                           return self._execute_standard(initial_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 185, in _execute_standard                                                 raise e
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 169, in _execute_standard                                                 result = current_node.execute(state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/nodes/generate_answer_node.py", line 129, in execute                                                  answer = chain.invoke({"question": user_prompt})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 2878, in invoke                                                             input = context.run(step.invoke, input, config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/output_parsers/base.py", line 192, in invoke                                                         return self._call_with_config(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1785, in _call_with_config                                                  context.run(
  File "/venv/lib/python3.12/site-packages/langchain_core/runnables/config.py", line 397, in call_func_with_variable_args                                      return func(input, **kwargs)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/output_parsers/base.py", line 193, in <lambda>                                                       lambda inner_input: self.parse_result([Generation(text=inner_input)]),
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/load/serializable.py", line 113, in __init__                                                         super().__init__(*args, **kwargs)
  File "/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 341, in __init__                                                                         raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for Generation
text
  str type expected (type=type_error.str)

To Reproduce Run the example: https://github.com/ScrapeGraphAI/Scrapegraph-ai/blob/main/examples/openai/smart_scraper_schema_openai.py

I also tried to update to the latest langchain, but that didn’t help. It could be langchain-related, as I can see similar issues (but not exactly the same) on their issues list. Or it’s just a mix of pydantic versions.

Expected behavior It should succeed.

Screenshots

Desktop (please complete the following information):

  • OS: MacOS
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 62 (25 by maintainers)

Most upvoted comments

hi, we will fix in the next hours

@rjbks @VinciGit00 I was able to fix the error for both ChatOpenAI and GoogleGenAI (tested on gemini-1.5-pro).

I will open a pull request for this error. At the end it was related with the initialisation problem.

v1.18.1 do not contain the fixed version

It is present in v1.19.0-beta.1+ If it works there it should be fine

Can confirm the same error persists in 1.16.0:

Traceback (most recent call last):
  File "/workspaces/Marsala/marsala.py", line 47, in <module>
    result = smart_scraper_graph.run()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/scrapegraphai/graphs/smart_scraper_graph.py", line 114, in run
    self.final_state, self.execution_info = self.graph.execute(inputs)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 263, in execute
    return self._execute_standard(initial_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 184, in _execute_standard
    raise e
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 168, in _execute_standard
    result = current_node.execute(state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/scrapegraphai/nodes/generate_answer_node.py", line 134, in execute
    answer = chain.invoke({"question": user_prompt})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 2878, in invoke
    input = context.run(step.invoke, input, config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 5092, in invoke
    return self.bound.invoke(
           ^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 277, in invoke
    self.generate_prompt(
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 777, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 634, in generate
    raise e
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 624, in generate
    self._generate_with_cache(
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 846, in _generate_with_cache
    result = self._generate(
             ^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/langchain_openai/chat_models/base.py", line 652, in _generate
    response = self.root_client.beta.chat.completions.parse(**payload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/openai/resources/beta/chat/completions.py", line 118, in parse
    response_format=_type_to_response_format(response_format),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.cache/pypoetry/virtualenvs/marsala-7QkPqEIU-py3.12/lib/python3.12/site-packages/openai/lib/_parsing/_completions.py", line 255, in type_to_response_format_param
    raise TypeError(f"Unsupported response_format type - {response_format}")
TypeError: Unsupported response_format type - <class '__main__.Companies'>

Code:

import scrapegraphai
import json
from typing import List
#from pydantic import BaseModel, Field
from langchain_core.pydantic_v1 import BaseModel, Field
from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict
from scrapegraphai.utils import prettify_exec_info
from scrapegraphai.graphs import SmartScraperGraph, OmniScraperGraph

class Settings(BaseSettings):
    openai_api_key: SecretStr

    model_config = SettingsConfigDict(env_file='.env', env_file_encoding='utf-8')

class Company(BaseModel):
    company: str = Field(description="Company name")
    description: str = Field(description="Company description")
    email: str = Field(description="Company email")

class Companies(BaseModel):
    companies: List[Company]

if __name__ == "__main__":
    # Define the configuration for the scraping pipeline
    settings = Settings()

    graph_config = {
        "llm": {
            "api_key": settings.openai_api_key.get_secret_value(),
            "model": "openai/gpt-4o-mini",
        },
        "verbose": True,
        "headless": True,
    }

    # Create the SmartScraperGraph instance
    smart_scraper_graph = SmartScraperGraph(
        prompt="Find some information about what the company does, the name and a contact email.",
        source="https://scrapegraphai.com/",
        schema=Companies,
        config=graph_config
    )

    # Run the pipeline
    result = smart_scraper_graph.run()
    print(result)
    #print(json.dumps(result, indent=4))

    graph_exec_info = smart_scraper_graph.get_execution_info()
    print(prettify_exec_info(graph_exec_info))

Yeah but langchain is fine with pydantic v2, and I tested without success pydantic v1.

However, I investigated and I found that there is an issue inGenerateAnswerNode: both self.llm_model.with_structured_output and JsonOutputParser are used, but with_structured_output already adds an output parser: https://github.com/langchain-ai/langchain/blob/fabd3295fabb4c79fedb4dbbe725a308658ef8d8/libs/partners/openai/langchain_openai/chat_models/base.py#L1414C25-L1414C36

So it’s effectively trying to parse a Pydantic object.

So you can either ask the llm class to return a structured output: https://github.com/ScrapeGraphAI/Scrapegraph-ai/blame/a96617d6f88f7370ecb7c58d0a62d3bdc0d80b31/scrapegraphai/nodes/generate_answer_node.py#L103 Or provide an output parser: https://github.com/ScrapeGraphAI/Scrapegraph-ai/blame/a96617d6f88f7370ecb7c58d0a62d3bdc0d80b31/scrapegraphai/nodes/generate_answer_node.py#L133C47-L133C47

(side note: it seems that json_schema mode is only implemented for openai chats, not for mistral or others - but langchain will ignore it silently: https://github.com/ScrapeGraphAI/Scrapegraph-ai/blame/a96617d6f88f7370ecb7c58d0a62d3bdc0d80b31/scrapegraphai/nodes/generate_answer_node.py#L100)

I get a similar error:

--- Executing GenerateAnswer Node ---
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/llm_scraper/scrape.py", line 182, in <module>
    main()
  File "/llm_scraper/scrape.py", line 177, in main
    for url in _iterate_provider_list(provider_id, base_graph_config):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/llm_scraper/scrape.py", line 125, in _iterate_provider_list
    result = script_creator_graph.run()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/smart_scraper_graph.py", line 114, in run
    self.final_state, self.execution_info = self.graph.execute(inputs)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 263, in execute
    return self._execute_standard(initial_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 185, in _execute_standard
    raise e
  File "/venv/lib/python3.12/site-packages/scrapegraphai/graphs/base_graph.py", line 169, in _execute_standard
    result = current_node.execute(state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/scrapegraphai/nodes/generate_answer_node.py", line 129, in execute
    answer = chain.invoke({"question": user_prompt})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 2878, in invoke
    input = context.run(step.invoke, input, config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 5092, in invoke
    return self.bound.invoke(
           ^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 276, in invoke
    self.generate_prompt(
  File "/venv/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 776, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 633, in generate
    raise e
  File "/venv/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 623, in generate
    self._generate_with_cache(
  File "/venv/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 845, in _generate_with_cache
    result = self._generate(
             ^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/langchain_openai/chat_models/base.py", line 629, in _generate
    response = self.root_client.beta.chat.completions.parse(**payload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/openai/resources/beta/chat/completions.py", line 118, in parse
    response_format=_type_to_response_format(response_format),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/openai/lib/_parsing/_completions.py", line 245, in type_to_response_format_param
    raise TypeError(f"Unsupported response_format type - {response_format}")
TypeError: Unsupported response_format type - <class 'llm_scraper.models.bronze.UrlsList'>

Did you manage to reproduce using the example?

@rjbks https://github.com/ScrapeGraphAI/Scrapegraph-ai/pull/664 #664

After all you have discovered, the error is still present in both OpenAi and Google versions?

There seem to be 2, possibly unrelated, issues here. First is boolean json representations throw an error as they are represented in the python syntax (title case like True instead of true). Second seems to be something that is most likely specific to either the max_results number of pages searched, or with a specific chunk of data extracted from a specific page in those search results in my code above. These 2 errors, and not the specific error the OP started the thread with are the issues here. I suppose this belongs in another issue/thread. Change the max_results config value to 20 and see what happens with my specific query/code above. It throws for me consistently.

After all you have discovered, the error is still present in both OpenAi and Google versions?

It is working for me in 1.19beta with OpenAI

In Version 1.16.0b2 is still broken.

There are two possible fix for this error, I tried them locally and they both work.

FIX 1 GenerateAnswerNode - Line 89

 if self.node_config.get("schema", None) is not None:

            # if isinstance(self.llm_model, (ChatOpenAI, ChatMistralAI)):
            #     self.llm_model = self.llm_model.with_structured_output(
            #         schema = self.node_config["schema"],
            #         method="json_schema")
            # else: 
            output_parser = JsonOutputParser(pydantic_object=self.node_config["schema"])

        else:
            output_parser = JsonOutputParser()

The fix is commenting out the with_strctured_output part. Commenting out this part I tested smart_scraper_schema_openai both with pydantic and langchain_core.pydantic_v1 and it works.

FIX 2 GenerateAnswerNode L89

 if self.node_config.get("schema", None) is not None:

            if isinstance(self.llm_model, (ChatOpenAI, ChatMistralAI)):
                self.llm_model = self.llm_model.with_structured_output(
                    schema = self.node_config["schema"],
                    method="json_schema")
            else: 
                output_parser = JsonOutputParser(pydantic_object=self.node_config["schema"])
                format_instructions = output_parser.get_format_instructions()

        else:
            output_parser = JsonOutputParser()
            format_instructions = output_parser.get_format_instructions()

Firstly, format instructions are not necessary anymore when using with_strcutured_output, so the instruction must be moved inside.

Than every invocation of the llm should be modified as follows:

prompt = PromptTemplate(
                template=template_no_chunks_prompt ,
                input_variables=["question"],
                partial_variables={"context": doc})
chain =  prompt | self.llm_model # | output_parser
answer = chain.invoke({"question": user_prompt})

removed the output parser and the format instructions.

With this solution there are 3 problems:

  • Doubling all the prompts to have versions with and without format instructions
  • It needs a custom parser at the end to use the answer, output parser from langchain cannot be used with this mode, thus the answer is a well formatted string that is missing all the json annotation
projects=[Project(title='Rotary Pendulum RL', description='Open Source project aimed at controlling a real life rotary pendulum using RL algorithms.'), Project(title='DQN Implementation from scratch', description='Developed a Deep Q-Network algorithm to train a simple and double pendulum.'), Project(title='Multi Agents HAED', description='University project which focuses on simulating a multi-agent system to perform environment mapping. Agents, equipped with sensors, explore and record their surroundings, considering uncertainties in their readings.'), Project(title='Wireless ESC for Modular Drones', description='Modular drone architecture proposal and proof of concept. The project received maximum grade.')]
  • It only works when creating the schema with pydantic, langchain_core.pydantic_v1 do not work here.

Hi, I wanted to let you know that this comment is still valid for version 1.17.0, the latest one on branch pre/beta.

Thanks for reporting this. I’ve had similar issues myself while working on a side project.

I managed to fix my problem by tinkering with the system prompt. I’ll see if I can do the same here, and let you know.