pydantic: Recurse KeyError when using several BaseModels

Initial Checks

I confirm that I’m using Pydantic V2

Description

The following code is working well with pydantic 2.3, but after upgrading it to 2.4 it fails with the following error:

Traceback (most recent call last):
  File "/home/ahoorelbeke/generator_v1_detailed/src/generator_source/parameters/parameters_equipment_design.py", line 23, in <module>
    class E(BaseModel):
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 184, in __new__
    complete_model_class(
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 495, in complete_model_class
    schema = apply_discriminators(simplify_schema_references(schema))
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 519, in simplify_schema_references
    schema = walk_core_schema(schema, count_refs)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 439, in walk_core_schema
    return f(schema, _dispatch)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 515, in count_refs
    recurse(state['definitions'][ref], count_refs)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 215, in walk
    return f(schema, self._walk)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 503, in count_refs
    return recurse(s, count_refs)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 218, in _walk
    schema = self._schema_type_to_method[schema['type']](schema, f)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 227, in _handle_other_schemas
    schema['schema'] = self.walk(sub_schema, f)  # type: ignore
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 215, in walk
    return f(schema, self._walk)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 503, in count_refs
    return recurse(s, count_refs)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 218, in _walk
    schema = self._schema_type_to_method[schema['type']](schema, f)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 362, in handle_model_fields_schema
    replaced_field['schema'] = self.walk(v['schema'], f)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 215, in walk
    return f(schema, self._walk)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 515, in count_refs
    recurse(state['definitions'][ref], count_refs)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 218, in _walk
    schema = self._schema_type_to_method[schema['type']](schema, f)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 227, in _handle_other_schemas
    schema['schema'] = self.walk(sub_schema, f)  # type: ignore
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 215, in walk
    return f(schema, self._walk)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 503, in count_refs
    return recurse(s, count_refs)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 218, in _walk
    schema = self._schema_type_to_method[schema['type']](schema, f)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 362, in handle_model_fields_schema
    replaced_field['schema'] = self.walk(v['schema'], f)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 215, in walk
    return f(schema, self._walk)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 503, in count_refs
    return recurse(s, count_refs)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 218, in _walk
    schema = self._schema_type_to_method[schema['type']](schema, f)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 227, in _handle_other_schemas
    schema['schema'] = self.walk(sub_schema, f)  # type: ignore
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 215, in walk
    return f(schema, self._walk)
  File "/home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 515, in count_refs
    recurse(state['definitions'][ref], count_refs)
KeyError: '__main__.B:94275587272912'

Example Code

from pydantic import BaseModel


class A(BaseModel):
    a: int = None


class B(BaseModel):
    a_1: A = None
    a_2: A = None


class C(BaseModel):
    a: A = None
    b: B = None


class D(BaseModel):
    c_1: C = None
    c_2: C = None


class E(BaseModel):
    c: C

Python, Pydantic & OS Version

        pydantic version: 2.4.0
        pydantic-core version: 2.10.0
          pydantic-core build: profile=release pgo=true
                 install path: /home/ahoorelbeke/venv/lib/python3.10/site-packages/pydantic
               python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
                     platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
             related packages: typing_extensions-4.7.1

About this issue

Original URL
State: closed
Created 9 months ago
Reactions: 13
Comments: 15 (6 by maintainers)

Commits related to this issue

Remove schema building caches Fixes https://github.com/pydantic/pydantic/issues/7611 — committed to pydantic/pydantic by adriangb 9 months ago
✅ Add test for the case from #7611 — committed to pydantic/pydantic by lig 9 months ago
Add negative dependency for Pydantic 2.4.0 https://github.com/pydantic/pydantic/issues/7611 — committed to object-Object/HexMod by object-Object 9 months ago
✅ Add test for the case from #7611 (#7619) Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> — committed to pydantic/pydantic by lig 9 months ago
Add negative dependency for Pydantic 2.4.0 https://github.com/pydantic/pydantic/issues/7611 — committed to hexdoc-dev/hexdoc by object-Object 9 months ago
Squash all pre-refactor hexdoc commits Squashed commit of the following: commit 04f86af56dcae682a70ef31f97e9ab99e65af40a Author: object-Object <object@objectobject.ca> Date: Fri Nov 3 02:52:49 202... — committed to object-Object/HexMod by object-Object 7 months ago

Most upvoted comments

We are planning to release v2.4.1 with the current fix on main and then investigate further the iceberg case above. Hopefully this mitigates the issue for the majority of folks.

davidhewitt on Sep 26, 2023

v2.4.2 is working @adriangb, thank you!

nmichlo on Sep 28, 2023

@nmichlo can you try main (or 2.4.2 once it comes out)?

adriangb on Sep 27, 2023

This should be fixed on main now (38bc2da1a68a71c9f36c61c000bb95ebc77764ef). @hoorelbeke-jimmy, @Fokko could you try installing from main and see if it fixes things?

adriangb on Sep 26, 2023

This still seems broken for me in v2.4.1, reverting to v2.3.0 solved things.

nmichlo on Sep 27, 2023

I verified now from my end ( via pip install of the latest version 2.4.1 - not via git@main).

It works flawlessly!

😍

Thanks!

kefahi on Sep 26, 2023

his should be fixed on main now (38bc2da).

It works for me now, aiogram fixed by this update. Thanks! Waiting for the release in pypi.

JrooTJunior on Sep 26, 2023

Still seeing issues: https://github.com/apache/iceberg/pull/8646

I’m going to take a look at this.

adriangb on Sep 26, 2023

This should be fixed on main now (38bc2da). @hoorelbeke-jimmy, @Fokko could you try installing from main and see if it fixes things?

My snippet works when installing pydantic from main

hoorelbeke-jimmy on Sep 26, 2023