pydantic: Skip invalid items of collections instead of raising ValidationError
Checks
- I added a descriptive title to this issue
- I have searched (google, github) for similar issues and couldn’t find anything
- I have read and followed the docs and still think this feature/change is needed
-  After submitting this, I commit to one of:
- Look through open issues and helped at least one other person
- Hit the “watch” button on this repo to receive notifications and I commit to help at least 2 people that ask questions in the future
- Implement a Pull Request for a confirmed bug
 
Feature Request
When parsing data, you may encounter some invalid items in a list. In some cases, it’s not worth to through away all of the data because of just one invalid item.
What I need is to skip invalid items if a list and leave only valid ones.
Sure, this can be done using @validator(pre=True), but this needs to be default behaviour of my models and to accomplish this I will basically have to write custom validator for every single list by hand, which is not very handy.
Related issues
Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":
             pydantic version: 1.6.1
            pydantic compiled: True
                 install path: /Users/rocky/.local/share/virtualenvs/.../lib/python3.6/site-packages/pydantic
               python version: 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)  [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
                     platform: Darwin-20.1.0-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']
Here’s how I expect it to work:
from pydantic import BaseModel
from pydantic.types import OptionalList
class Model(BaseModel):
    a: OptionalList[int]
m = Model(a=[1, '2', '?'])
assert m.a == [1, 2]
assert m.__skipped__ == [{'loc': ('a', 2), 'msg': 'value is not a valid integer', 'type': 'type_error.integer'}]
# skipped list should be returned by `validate_model()` just like errors
...
Things to think of:
- Support ofconlist,consetetc… and interaction with them
- Default values instead of invalid skipped ones
Another possible syntax:
a: List[int] = Field(skip_invalid=True)
class Config:
    skip_invalid_collections_items = True
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 22
- Comments: 19 (11 by maintainers)
Here’s my temporary implementation. It works as expected with
GenericModeland generates schema identical totyping.List.Is there any example anywhere of using this
on_error='omit'in v2?In case anyone is waiting for feedback on this,
LenientListas implemented here by Bobronium works for me in a complex schema, with pydantic 1.10.2. I don’t know the internals of pydantic enough to give a proper code review though.If it helps, here are a handful of basic pytest tests I’ve written for my own peace of mind (seeing as LenientList relies on pydantic internals, I want to make sure it doesn’t break at the next pydantic upgrade).
@samuelcolvin that doesn’t expose a way to retrieve the errors afterward does it? I implemented a slight tweak to @adriangb’s very good suggestion here: https://gist.github.com/dmontagu/7f0cef76e5e0e04198dd608ad7219573, this creates a type
LenientListwhich is a subclass ofList[T]and adds an attribute callederrorsthat lets you retrieve the validation errors, and has a methodwith_errors()that returns the items and errors in the order corresponding to validation.The implementation is a little ugly on the inside, but perhaps someone will find it useful.
Either way, @adriangb’s code is another good example of how to use annotations to create pydantic core schemas, and my gist above is (in my opinion) a good example of how to create a pydantic core schema for a custom generic type.
It’s also supported out of the box in pydantic-core, see a test example here and the schema to configure it here.
I implemented this months ago thinking of this issue, but forgot to reference it and we haven’t added support in python yet.
I have some good news, after thinking about this for a bit I think this not only works in V2 but is also super powerful and re-usable:
This passes all of the tests in https://github.com/pydantic/pydantic/issues/2274#issuecomment-1432916116
Obviously you may want slightly different behavior, but I think this general template can be adapted.
No hurry, some time before the end of March would be great.
collecting errors should become possible in V2 with
context, see here.This should be possible in V2 with
on_error='omit'but we should make sure it’s actually possible before release.