pydantic: Memory leak after updating to Pydantic v1.9.0

Checks

  • I added a descriptive title to this issue
  • I have searched (google, github) for similar issues and couldn’t find anything
  • I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

$ python -c "import pydantic.utils; print(pydantic.utils.version_info())"
             pydantic version: 1.9.0
            pydantic compiled: True
                 install path: /opt/pydist/.venv/lib/python3.10/site-packages/pydantic
               python version: 3.10.1 (main, Dec 21 2021, 09:01:08) [GCC 10.2.1 20210110]
                     platform: Linux-5.10.93-x86_64-with-glibc2.31
     optional deps. installed: ['dotenv', 'typing-extensions']

As a heads-up we are seeing a memory leak in production after upgrading to Pydantic version 1.9.0. Our application is a large FastAPI based API service (fastapi 0.73.0, uvicorn 0.17.4). The application uses Pydantic BaseModels, GenericModels and pydantic.dataclasses.dataclasses for FastAPI response/request data models. It also uses Pydantic models for some internal data validators. We deployed a single change to production, which changed the version of Pydantic from 1.8.2 to 1.9.0. After that we started to observe abnormal memory usage. The memory usage increased until the version upgrade got reverted back to the older 1.8.2 version (no other change again). After deploying the revert, the memory usage patterns went back to normal.

Unfortunately I haven’t been able to reproduce the issue locally (it is an effort to simulate the prod-like load to get the leak to show up). So I can not yet add further information from isolating it or information from eg. memory analyzers. I tried to also look into the changelog and diff but its so big release that I can not spot anything obvious that may cause this. I have attached below a picture from our production metrics about the abnormal behaviour after using v1.9.0. We have never observed such a memory issue until now when upgrading to v1.9.0.

Thank you all for the awesome work and the great library! I wish I would have more information about the leak for the ticket.

Näyttökuva 2022-2-18 kello 8 48 24 Above shows the different deploys in colors. The orange one is where the Pydantic 1.9.0 update was running until it was reverted due to the memory issue. We couldn't have run it any longer as it was close at hitting the Kubernetes pod memory limits at that point.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 2
  • Comments: 28 (16 by maintainers)

Most upvoted comments

Well after hours of hunting, the good news is I think I’ve found a memory leak, the bad news is it’s in python and I can’t see how it’s specific to v1.9.

https://github.com/python/cpython/issues/92810

😞

Yes I think so.

Also:

  • it’s probably not used by that many people, which might explain why not everyone sees this
  • It definitely changed in v1.9
  • there’s an obvious way for memory to build up via _assigned_parameters

I think I’ll put a limit on the size of those two cache dicts just in case.