pydantic: npt.NDArray with `arbitrary_types_allowed=True` raises `pydantic.errors.PydanticSchemaGenerationError`

Initial Checks

I confirm that I’m using Pydantic V2 installed directly from the main branch, or equivalent

Description

I’m having the following BaseModel that was previously working with v1:

CustomNDArrayType: TypeAlias = Union[npt.NDArray, List[npt.NDArray]]

class SomeModel(BaseModel):
    data: Optional[CustomNDArrayType]

    model_config = ConfigDict(
        arbitrary_types_allowed=True,
        extra="forbid",
        validate_assignment=True,
    )

but in v2 raises the following:

pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for numpy.ndarray[typing.Any, numpy.dtype[+ScalarType]]. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.

If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.
For further information visit https://errors.pydantic.dev/2.0.2/u/schema-for-unknown-type

Strange thing is, that setting arbitrary_types_allowed=True doesn’t ignore the error at all. Any ideas how to deal with this?

Example Code

CustomNDArrayType: TypeAlias = Union[npt.NDArray, List[npt.NDArray]]

class SomeModel(BaseModel):
    data: Optional[CustomNDArrayType]

    model_config = ConfigDict(
        arbitrary_types_allowed=True,
        extra="forbid",
        validate_assignment=True,
    )

Python, Pydantic & OS Version

pydantic version: 2.0.2
        pydantic-core version: 2.1.2 release build profile
                 install path: /Users/panagiotisvardanis/miniforge3/envs/mac-tensorflow/lib/python3.8/site-packages/pydantic
               python version: 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:01:13)  [Clang 14.0.6 ]
                     platform: macOS-13.4.1-arm64-arm-64bit
     optional deps. installed: ['typing-extensions']

Selected Assignee: @adriangb

About this issue

Original URL
State: closed
Created a year ago
Reactions: 1
Comments: 15 (8 by maintainers)

Most upvoted comments

What about 3.8.17?

I get the same error

pvardanis on Jul 6, 2023

If #6484 gets merged — and it might still need some changes or otherwise get rejected — you’d be able to do this (taken from #6484 (comment)):

from typing import Union, List, Optional, Any

import numpy as np
import numpy.typing as npt
from typing_extensions import TypeAlias, Annotated

from pydantic import BaseModel, GetPydanticSchema, InstanceOf

VALIDATE_NDARRAY_INSTANCES = False
if VALIDATE_NDARRAY_INSTANCES:
    PydanticNDArray: TypeAlias = Annotated[npt.NDArray, GetPydanticSchema.for_type(InstanceOf[np.ndarray])]
else:
    PydanticNDArray: TypeAlias = Annotated[npt.NDArray, GetPydanticSchema.for_type(Any)]

CustomNDArrayType: TypeAlias = Union[PydanticNDArray, List[PydanticNDArray]]


class SomeModel(BaseModel):
    data: Optional[CustomNDArrayType]

print(SomeModel(data=None))
# data=None

print(SomeModel(data=np.array([1, 2, 3])))
# data=array([1, 2, 3])

print(SomeModel(data=[np.array([1, 2, 3])]))
# data=[array([1, 2, 3])]

print(SomeModel(data=[1]))
# If VALIDATE_NDARRAY_INSTANCES is False:
#> data=[1]  

# If VALIDATE_NDARRAY_INSTANCES is True:
"""
pydantic_core._pydantic_core.ValidationError: 2 validation errors for SomeModel
data.is-instance[ndarray]
  Input should be an instance of ndarray [type=is_instance_of, input_value=[1], input_type=list]
    For further information visit https://errors.pydantic.dev/2.1.2/v/is_instance_of
data.list[is-instance[ndarray]].0
  Input should be an instance of ndarray [type=is_instance_of, input_value=1, input_type=int]
    For further information visit https://errors.pydantic.dev/2.1.2/v/is_instance_of
"""

Note that I have included the VALIDATE_NDARRAY_INSTANCES flag since, in V1, when a field required allow_arbitrary_types=True it would be treated as Any by pydantic, whereas in V2, it is validated using isinstance. So the above code snippet can be configured to get either the v1-style (no validation) or v2-style (validate that the input is an instance of np.ndarray) validation behavior.

Thanks a lot! When should I expect this to be released in a new Pydantic version, except main?

pvardanis on Jul 7, 2023

We check that the type annotation is a type: https://github.com/pydantic/pydantic/blob/8466f78da6b42d404fa721be6b3f57061b99cb30/pydantic/_internal/_generate_schema.py#L579C2-L579C2

And on < 3.9 that numpy type is not in fact a type: https://github.com/numpy/numpy/blob/1ed0bfd3a42bf0c8a0060684645ba97941778006/numpy/_typing/_generic_alias.py#L245.

I’ll make a PR with a better error message, but I’d also suggest submitting a PR to numpy to try to make that pass isinstance(..., type)

adriangb on Jul 6, 2023

Ok, I can confirm on 3.8.16 I get the error.

hramezani on Jul 6, 2023